fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002258720>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002258720>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:108]: failed to get service managed identity
Unexpected error:
    <*exported.ResponseError | 0xc0017f8e10>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:108]: failed to get service managed identity
Unexpected error:
    <*exported.ResponseError | 0xc0017f8e10>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013bd140>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xj578sgpwt6l" to finish creating: GET https://rp.j2660096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "name": "0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "status": "Failed",
      "startTime": "2026-10-02T00:27:55.227836447Z",
      "endTime": "2026-10-02T00:47:04.606186902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013bd140>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xj578sgpwt6l" to finish creating: GET https://rp.j2660096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "name": "0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "status": "Failed",
      "startTime": "2026-10-02T00:27:55.227836447Z",
      "endTime": "2026-10-02T00:47:04.606186902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:130]: HCP cluster rg-candidate-4-20-vgwv7thnmvhb/cluster-candidate-4-20-mvn65b should provision
Unexpected error:
    <*fmt.wrapError | 0xc001036300>: 
    failed to create HCP cluster cluster-candidate-4-20-mvn65b: failed waiting for cluster="cluster-candidate-4-20-mvn65b" in resourcegroup="rg-candidate-4-20-vgwv7thnmvhb" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "name": "c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "status": "Failed",
      "startTime": "2026-10-01T23:41:46.834189313Z",
      "endTime": "2026-10-02T00:10:52.697527079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.40 is Partial (want Completed), started 1m39s ago; version 4.20.40 is Partial (want Completed), started 4m49s ago; version 4.20.40 is Partial (want Completed), started 10m52s ago; version 4.20.40 is Partial (want Completed), started 14m44s ago; version 4.20.40 is Partial (want Completed), started 19m16s ago; version 4.20.40 is Partial (want Completed), started 21m37s ago; version 4.20.40 is Partial (want Completed), started 23m41s ago; version 4.20.40 is Partial (want Completed), started 24m16s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:130]: HCP cluster rg-candidate-4-20-vgwv7thnmvhb/cluster-candidate-4-20-mvn65b should provision
Unexpected error:
    <*fmt.wrapError | 0xc001036300>: 
    failed to create HCP cluster cluster-candidate-4-20-mvn65b: failed waiting for cluster="cluster-candidate-4-20-mvn65b" in resourcegroup="rg-candidate-4-20-vgwv7thnmvhb" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "name": "c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "status": "Failed",
      "startTime": "2026-10-01T23:41:46.834189313Z",
      "endTime": "2026-10-02T00:10:52.697527079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.40 is Partial (want Completed), started 1m39s ago; version 4.20.40 is Partial (want Completed), started 4m49s ago; version 4.20.40 is Partial (want Completed), started 10m52s ago; version 4.20.40 is Partial (want Completed), started 14m44s ago; version 4.20.40 is Partial (want Completed), started 19m16s ago; version 4.20.40 is Partial (want Completed), started 21m37s ago; version 4.20.40 is Partial (want Completed), started 23m41s ago; version 4.20.40 is Partial (want Completed), started 24m16s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:182]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000fb92c0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-d4n779" in resourcegroup="rg-np-version-upgrade-d4n779-msk5sf2n8dpt" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f",
      "name": "d667a581-6512-4aad-a4a8-0592dbc7103f",
      "status": "Failed",
      "startTime": "2026-10-02T00:18:31.430555435Z",
      "endTime": "2026-10-02T00:37:32.349627981Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:182]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000fb92c0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-d4n779" in resourcegroup="rg-np-version-upgrade-d4n779-msk5sf2n8dpt" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f",
      "name": "d667a581-6512-4aad-a4a8-0592dbc7103f",
      "status": "Failed",
      "startTime": "2026-10-02T00:18:31.430555435Z",
      "endTime": "2026-10-02T00:37:32.349627981Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T23:46:43Z
  Ended:       2026-10-01T23:52:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1600128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t69mr53lpaagaj8ndpfbdmi4pj01tkd-j7t7p3j9q1q0x2g", pod="machine-approver-74799c449d-qwlv2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t69mr53lpaagaj8ndpfbdmi4pj01tkd-j7t7p3j9q1q0x2g/machine-approver-74799c449d-qwlv2 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 1200.000s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 1200.000s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T23:10:36Z
  Ended:       2026-10-01T23:23:39Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9983616-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T23:11:38Z
  Ended:       2026-10-01T23:25:38Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9983616-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-10-01T23:12:38Z
  Ended:       2026-10-01T23:23:39Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9983616-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000121720>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-79c5f4ztf9wb" to finish creating: GET https://rp.j0967040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6667e395-9474-42fe-b0f1-4ba196079157
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6667e395-9474-42fe-b0f1-4ba196079157",
      "name": "6667e395-9474-42fe-b0f1-4ba196079157",
      "status": "Failed",
      "startTime": "2026-10-01T22:21:23.12653074Z",
      "endTime": "2026-10-01T22:40:23.030212068Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-glx29-lkqgm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-79c5f4ztf9wb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-glx29-lkqgm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-glx29-lkqgm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-79c5f4ztf9wb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-glx29-lkqgm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000121720>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-79c5f4ztf9wb" to finish creating: GET https://rp.j0967040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6667e395-9474-42fe-b0f1-4ba196079157
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6667e395-9474-42fe-b0f1-4ba196079157",
      "name": "6667e395-9474-42fe-b0f1-4ba196079157",
      "status": "Failed",
      "startTime": "2026-10-01T22:21:23.12653074Z",
      "endTime": "2026-10-01T22:40:23.030212068Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-glx29-lkqgm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-79c5f4ztf9wb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-glx29-lkqgm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-glx29-lkqgm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-79c5f4ztf9wb--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-glx29-lkqgm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq"
Unexpected error:
    <*fmt.wrapError | 0xc0010a7280>: 
    failed to create NodePool np-xxkkqq: failed waiting for nodepool="np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq" in resourcegroup="rg-hs-pre--9kx2vcffdvgg" to finish creating: GET https://rp.j0967040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "name": "9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "status": "Failed",
      "startTime": "2026-10-01T22:20:04.161467493Z",
      "endTime": "2026-10-01T22:39:14.657703099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq"
Unexpected error:
    <*fmt.wrapError | 0xc0010a7280>: 
    failed to create NodePool np-xxkkqq: failed waiting for nodepool="np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq" in resourcegroup="rg-hs-pre--9kx2vcffdvgg" to finish creating: GET https://rp.j0967040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "name": "9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "status": "Failed",
      "startTime": "2026-10-01T22:20:04.161467493Z",
      "endTime": "2026-10-01T22:39:14.657703099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001422540>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-nr76gskcfsht" to finish creating: GET https://rp.j8617344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "name": "d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "status": "Failed",
      "startTime": "2026-10-01T21:37:37.328442226Z",
      "endTime": "2026-10-01T22:06:41.36782171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001422540>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-nr76gskcfsht" to finish creating: GET https://rp.j8617344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "name": "d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "status": "Failed",
      "startTime": "2026-10-01T21:37:37.328442226Z",
      "endTime": "2026-10-01T22:06:41.36782171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T21:56:38Z
  Ended:       2026-10-01T22:24:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T21:56:38Z
  Ended:       2026-10-01T22:24:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T21:59:25Z
  Ended:       2026-10-01T22:29:23Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a69f5d7-6f9f-4450-a9f7-f182ad9a23c6"
  Description: Pod ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T21:59:25Z
  Ended:       2026-10-01T22:29:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a69f5d7-6f9f-4450-a9f7-f182ad9a23c6"
  Description: Pod ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T22:11:26Z
  Ended:       2026-10-01T22:26:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T22:11:26Z
  Ended:       2026-10-01T22:26:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

time=2026-10-01T20:26:41.918Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-01T20:26:42.226Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4]"
time=2026-10-01T20:26:42.226Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-01T20:26:42.418Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-01T20:26:42.591Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-01T20:26:42.591Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-01T20:26:42.660Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-01T20:31:45.711Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-10-01T20:31:45.712Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"

time=2026-10-01T19:47:54.330Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-01T19:47:56.434Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-01T19:47:57.612Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=7fa4e9b0d5432f3733a8410e6c9eae279686c4d2104771439ec6c49754f75d3c portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2785792-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F7fa4e9b0d5432f3733a8410e6c9eae279686c4d2104771439ec6c49754f75d3c
time=2026-10-01T19:50:58.074Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/7fa4e9b0d5432f3733a8410e6c9eae279686c4d2104771439ec6c49754f75d3c/operationStatuses/08584107228086173802\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/cluster-suocoh45plnvg\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/ah-ci01-se-j2785792-1\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/ah-ci01-se-j2785792-1\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n                    \\\"message\\\": \\\"The vault name 'ah-ci01-se-j2785792-1' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-01T19:47:52.399Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-01T19:47:54.301Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-01T19:47:55.661Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=81d379c4f7c29ce92f1cd109cb00ebbe0457eec7d8a38ac495b54ae648addb81 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2503040%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F81d379c4f7c29ce92f1cd109cb00ebbe0457eec7d8a38ac495b54ae648addb81
time=2026-10-01T19:49:26.400Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2503040/providers/Microsoft.Resources/deployments/81d379c4f7c29ce92f1cd109cb00ebbe0457eec7d8a38ac495b54ae648addb81/operationStatuses/08584107228106955756\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2503040/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"BadRequest\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2503040/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j2503040\\\",\\r\\n            \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"BadRequest\\\\\\\",\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"The requested operation cannot be performed because the database account arohcpci01-rp-j2503040 is in the process of being created. Please wait until account creation completes and retry the operation.\\\\\\\\r\\\\\\\\nActivityId: deddb421-d755-4f62-b0ea-94301557d50e, Microsoft.Azure.Documents.Common/2.14.0\\\\\\\"\\\\r\\\\n}\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-10-01T19:47:48.734Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 description="Step infra\n  Kind: ARM\n  Template: templates/mgmt-infra.bicep\n  Parameters: configurations/mgmt-infra.tmpl.bicepparam"
time=2026-10-01T19:47:49.512Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2
time=2026-10-01T19:47:50.952Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 deployment=5be4ec9a5f0977ac9396f709d63fcfad01e98a314c6ed4279d81457cdf4fb342 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0582272-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5be4ec9a5f0977ac9396f709d63fcfad01e98a314c6ed4279d81457cdf4fb342
time=2026-10-01T19:50:21.739Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/5be4ec9a5f0977ac9396f709d63fcfad01e98a314c6ed4279d81457cdf4fb342/operationStatuses/08584107228152517010\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/ah-ci01-me-j0582272-2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-me-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/mgmt-kv-2awbxe73l63vi\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-mg-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/msi-kv-g2p3wnorghzau\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-mi-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 600.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 600.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000d76800>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-k45z8f" in resourcegroup="rg-np-version-upgrade-k45z8f-gbld887k6bmc" to finish creating: GET https://rp.j2806912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d30e2ef-e6cd-4682-9ed6-684519455e2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d30e2ef-e6cd-4682-9ed6-684519455e2c",
      "name": "4d30e2ef-e6cd-4682-9ed6-684519455e2c",
      "status": "Failed",
      "startTime": "2026-10-01T19:30:09.747572126Z",
      "endTime": "2026-10-01T19:49:10.285015393Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-k45z8f-gbld887k6bmc-np-upgrade-k4-fcb7102c/providers/Microsoft.Compute/virtualMachines/i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-k45z8f-gbld887k6bmc-np-upgrade-k4-fcb7102c/providers/Microsoft.Compute/virtualMachines/i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000d76800>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-k45z8f" in resourcegroup="rg-np-version-upgrade-k45z8f-gbld887k6bmc" to finish creating: GET https://rp.j2806912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d30e2ef-e6cd-4682-9ed6-684519455e2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d30e2ef-e6cd-4682-9ed6-684519455e2c",
      "name": "4d30e2ef-e6cd-4682-9ed6-684519455e2c",
      "status": "Failed",
      "startTime": "2026-10-01T19:30:09.747572126Z",
      "endTime": "2026-10-01T19:49:10.285015393Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-k45z8f-gbld887k6bmc-np-upgrade-k4-fcb7102c/providers/Microsoft.Compute/virtualMachines/i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-k45z8f-gbld887k6bmc-np-upgrade-k4-fcb7102c/providers/Microsoft.Compute/virtualMachines/i1u5g4i7q7n2v7a-npupgrade-4-20-h2z9m-sswnq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T19:40:45Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j2640384-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.90:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2640384/providers/microsoft.monitor/accounts/services-j2640384", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-77d869db9f-wr2wh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j2640384-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T19:40:45Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j2640384-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.90:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2640384/providers/microsoft.monitor/accounts/services-j2640384", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-77d869db9f-wr2wh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j2640384-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-dhvd4s" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000ed0460>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-dhvd4s: failed waiting for cluster="cp-ystream-upgrade-5-1-dhvd4s" in resourcegroup="rg-cp-ystream-upgrade-5-1-25w79q54vqwd" to finish creating: GET https://rp.j4153600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "name": "e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "status": "Failed",
      "startTime": "2026-10-01T18:01:36.572713358Z",
      "endTime": "2026-10-01T18:30:44.899304602Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-dhvd4s" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000ed0460>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-dhvd4s: failed waiting for cluster="cp-ystream-upgrade-5-1-dhvd4s" in resourcegroup="rg-cp-ystream-upgrade-5-1-25w79q54vqwd" to finish creating: GET https://rp.j4153600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "name": "e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "status": "Failed",
      "startTime": "2026-10-01T18:01:36.572713358Z",
      "endTime": "2026-10-01T18:30:44.899304602Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:36:27Z
  Ended:       2026-10-01T18:50:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qpxgq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:23:21Z
  Ended:       2026-10-01T18:53:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="71bdc953-4c3d-4f04-a433-73b33d9edd12"
  Description: Pod ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T18:23:21Z
  Ended:       2026-10-01T18:53:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="71bdc953-4c3d-4f04-a433-73b33d9edd12"
  Description: Pod ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:21:23Z
  Ended:       2026-10-01T18:49:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qpxgq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T18:32:21Z
  Ended:       2026-10-01T18:49:23Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qpxgq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd has not matched the expected number of replicas for longer than 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc00098e460>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-bkq9wzll9hld" to finish creating: GET https://rp.j8286336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8100d3ee-583f-4c7c-9e59-ea39fd0b9b3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8100d3ee-583f-4c7c-9e59-ea39fd0b9b3b",
      "name": "8100d3ee-583f-4c7c-9e59-ea39fd0b9b3b",
      "status": "Failed",
      "startTime": "2026-10-01T18:22:30.348651436Z",
      "endTime": "2026-10-01T18:41:32.581818657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw to exist\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw to exist\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc00098e460>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-bkq9wzll9hld" to finish creating: GET https://rp.j8286336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8100d3ee-583f-4c7c-9e59-ea39fd0b9b3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8100d3ee-583f-4c7c-9e59-ea39fd0b9b3b",
      "name": "8100d3ee-583f-4c7c-9e59-ea39fd0b9b3b",
      "status": "Failed",
      "startTime": "2026-10-01T18:22:30.348651436Z",
      "endTime": "2026-10-01T18:41:32.581818657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw to exist\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-6j7bw to exist\nMachine oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-bkq9wzll9hld--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-hwt4d-qlk22 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00102f760>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-56xdl2f56pkc" to finish creating: GET https://rp.j5666176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2329253-0de9-40f0-a7ca-fabe3f898d69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2329253-0de9-40f0-a7ca-fabe3f898d69",
      "name": "c2329253-0de9-40f0-a7ca-fabe3f898d69",
      "status": "Failed",
      "startTime": "2026-10-01T17:56:54.27865325Z",
      "endTime": "2026-10-01T18:15:59.07088369Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-56xdl2f56pkc--managed/providers/Microsoft.Compute/virtualMachines/x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-56xdl2f56pkc--managed/providers/Microsoft.Compute/virtualMachines/x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00102f760>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-56xdl2f56pkc" to finish creating: GET https://rp.j5666176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2329253-0de9-40f0-a7ca-fabe3f898d69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2329253-0de9-40f0-a7ca-fabe3f898d69",
      "name": "c2329253-0de9-40f0-a7ca-fabe3f898d69",
      "status": "Failed",
      "startTime": "2026-10-01T17:56:54.27865325Z",
      "endTime": "2026-10-01T18:15:59.07088369Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-56xdl2f56pkc--managed/providers/Microsoft.Compute/virtualMachines/x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-56xdl2f56pkc--managed/providers/Microsoft.Compute/virtualMachines/x2k8s0s2w4d6d9a-np-1-2rtc9-9wn9n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001483ce0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-8rn7xkjd78vr" to finish creating: GET https://rp.j5666176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dddd5f2b-92a0-447e-b74b-656cac310689
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dddd5f2b-92a0-447e-b74b-656cac310689",
      "name": "dddd5f2b-92a0-447e-b74b-656cac310689",
      "status": "Failed",
      "startTime": "2026-10-01T18:04:54.409641321Z",
      "endTime": "2026-10-01T18:23:59.28179146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-pq2md-zgb84: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8rn7xkjd78vr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-pq2md-zgb84 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-pq2md-zgb84: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8rn7xkjd78vr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-pq2md-zgb84 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001483ce0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-8rn7xkjd78vr" to finish creating: GET https://rp.j5666176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dddd5f2b-92a0-447e-b74b-656cac310689
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dddd5f2b-92a0-447e-b74b-656cac310689",
      "name": "dddd5f2b-92a0-447e-b74b-656cac310689",
      "status": "Failed",
      "startTime": "2026-10-01T18:04:54.409641321Z",
      "endTime": "2026-10-01T18:23:59.28179146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-pq2md-zgb84: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8rn7xkjd78vr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-pq2md-zgb84 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-pq2md-zgb84: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8rn7xkjd78vr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-pq2md-zgb84 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 96 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-mwtcx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-mwtcx has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-vvwct", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-vvwct has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-7bbsw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-7bbsw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-v4ftx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-v4ftx has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zgtdx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zgtdx has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-42mfl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-42mfl has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-lvnnt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-lvnnt has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-pghsk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-pghsk has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gxkml", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gxkml has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-jlzrr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-jlzrr has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-lvjnb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-lvjnb has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-4jj9w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-4jj9w has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-fpstv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-fpstv has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cgnq6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cgnq6 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-gxr4d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-gxr4d has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-98xtm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-98xtm has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-zc7gh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-zc7gh has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-qjtjc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-qjtjc has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-7dl97", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-7dl97 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-msjxj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-msjxj has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-lcvdj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-lcvdj has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cbjnh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cbjnh has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-wbsrk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-wbsrk has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-88hwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-88hwc has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-vpc4x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-vpc4x has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-rm7bz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-rm7bz has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-frwsn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-frwsn has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bmpv2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bmpv2 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-psq6v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-psq6v has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-59fkb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-59fkb has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-s9hxt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-s9hxt has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-d5ggn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-d5ggn has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-mz8hk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-mz8hk has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-rps7j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-rps7j has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gf894", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gf894 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-9zvzb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-9zvzb has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-29d68", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-29d68 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-g8db9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-g8db9 has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gn9th", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gn9th has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-4pklx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-4pklx has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-f8zbt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-f8zbt has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-qlxtj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-qlxtj has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zrnlh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zrnlh has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-6m7wq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-6m7wq has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-9rhtk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-9rhtk has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bl4jg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bl4jg has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-8pmcr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-8pmcr has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-9nxwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-9nxwc has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-9f96s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-9f96s has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-mmp68", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-mmp68 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-d4nqn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-d4nqn has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-c5nnh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-c5nnh has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-t48z7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-t48z7 has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-q4bwx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-q4bwx has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-f9m5n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-f9m5n has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-xb9s5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-xb9s5 has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-jh25x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-jh25x has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-g9tw9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-g9tw9 has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-r6qng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-r6qng has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-74wcq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-74wcq has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-7lct6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-7lct6 has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zkbkk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zkbkk has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-v7m2p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-v7m2p has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-dh2wj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-dh2wj has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-9frwj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-9frwj has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bvxz5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bvxz5 has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-hfm7f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-hfm7f has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-wjp8n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-wjp8n has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-n5vmn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-n5vmn has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-d8lx7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-d8lx7 has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-dlhhb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-dlhhb has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-k69bw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-k69bw has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-kczlx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-kczlx has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zdgk7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zdgk7 has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-rvmlf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-rvmlf has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gmfc6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gmfc6 has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-lddj8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-lddj8 has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-xm6fg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-xm6fg has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-fn46n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-fn46n has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-58r86", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-58r86 has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-cfqmp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-cfqmp has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cpccl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cpccl has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-th4tc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-th4tc has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-2xsng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-2xsng has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-z8ldz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-z8ldz has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-96jbc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-96jbc has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cltv6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cltv6 has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-sfd6l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-sfd6l has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bgmnc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bgmnc has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-phv2m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-phv2m has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-2jlkd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-2jlkd has been in a non-ready state for longer than 5 minutes.
Firing 92:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-cf2v5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-cf2v5 has been in a non-ready state for longer than 5 minutes.
Firing 93:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-gmspn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-gmspn has been in a non-ready state for longer than 5 minutes.
Firing 94:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-7rkp9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-7rkp9 has been in a non-ready state for longer than 5 minutes.
Firing 95:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-2566w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-2566w has been in a non-ready state for longer than 5 minutes.
Firing 96:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-66jst", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-66jst has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:91]: failed to deploy customer infrastructure
Unexpected error:
    <*fmt.wrapError | 0xc0008d79c0>: 
    failed waiting for deployment "customer-infra-delayed-rbac-cluster" in resourcegroup="delayed-rbac-wx4qr5bhxpxx" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delayed-rbac-wx4qr5bhxpxx/providers/Microsoft.Resources/deployments/customer-infra-delayed-rbac-cluster/operationStatuses/08584107303732954713
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-usudlv6oegmxo' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:91]: failed to deploy customer infrastructure
Unexpected error:
    <*fmt.wrapError | 0xc0008d79c0>: 
    failed waiting for deployment "customer-infra-delayed-rbac-cluster" in resourcegroup="delayed-rbac-wx4qr5bhxpxx" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delayed-rbac-wx4qr5bhxpxx/providers/Microsoft.Resources/deployments/customer-infra-delayed-rbac-cluster/operationStatuses/08584107303732954713
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-usudlv6oegmxo' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0011bece0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9kw8hcr4wckp" to finish creating: GET https://rp.j8727040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a37a2f12-f810-4d95-b6f9-e4f90badf521
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a37a2f12-f810-4d95-b6f9-e4f90badf521",
      "name": "a37a2f12-f810-4d95-b6f9-e4f90badf521",
      "status": "Failed",
      "startTime": "2026-10-01T17:14:34.470579236Z",
      "endTime": "2026-10-01T17:33:43.504606544Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9kw8hcr4wckp--managed/providers/Microsoft.Compute/virtualMachines/n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9kw8hcr4wckp--managed/providers/Microsoft.Compute/virtualMachines/n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0011bece0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9kw8hcr4wckp" to finish creating: GET https://rp.j8727040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a37a2f12-f810-4d95-b6f9-e4f90badf521
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a37a2f12-f810-4d95-b6f9-e4f90badf521",
      "name": "a37a2f12-f810-4d95-b6f9-e4f90badf521",
      "status": "Failed",
      "startTime": "2026-10-01T17:14:34.470579236Z",
      "endTime": "2026-10-01T17:33:43.504606544Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9kw8hcr4wckp--managed/providers/Microsoft.Compute/virtualMachines/n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9kw8hcr4wckp--managed/providers/Microsoft.Compute/virtualMachines/n1x2c2o8g6n1p9y-arm64-vm-np-1-wtd85-cqmmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc00133e5e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-hn4smn" in resourcegroup="rg-np-version-upgrade-hn4smn-jsfvjxzg67gj" to finish creating: GET https://rp.j8727040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c8d2003-62e9-4366-9802-82ea1de6d2d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c8d2003-62e9-4366-9802-82ea1de6d2d5",
      "name": "2c8d2003-62e9-4366-9802-82ea1de6d2d5",
      "status": "Failed",
      "startTime": "2026-10-01T17:41:18.825800843Z",
      "endTime": "2026-10-01T18:00:23.360075645Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hn4smn-jsfvjxzg67gj-np-upgrade-hn-0c8b6b11/providers/Microsoft.Compute/virtualMachines/r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hn4smn-jsfvjxzg67gj-np-upgrade-hn-0c8b6b11/providers/Microsoft.Compute/virtualMachines/r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc00133e5e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-hn4smn" in resourcegroup="rg-np-version-upgrade-hn4smn-jsfvjxzg67gj" to finish creating: GET https://rp.j8727040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c8d2003-62e9-4366-9802-82ea1de6d2d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c8d2003-62e9-4366-9802-82ea1de6d2d5",
      "name": "2c8d2003-62e9-4366-9802-82ea1de6d2d5",
      "status": "Failed",
      "startTime": "2026-10-01T17:41:18.825800843Z",
      "endTime": "2026-10-01T18:00:23.360075645Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hn4smn-jsfvjxzg67gj-np-upgrade-hn-0c8b6b11/providers/Microsoft.Compute/virtualMachines/r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hn4smn-jsfvjxzg67gj-np-upgrade-hn-0c8b6b11/providers/Microsoft.Compute/virtualMachines/r1h6e6k4b1j0v4b-npupgrade-4-20-j2rcx-r7c5j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc00262f770>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc00262f770>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

time=2026-10-01T15:34:21.055Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.MgmtAgent resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> mgmtAgent.image.registry <nil>}:{<nil> mgmtAgent.image.repository <nil>}@{<nil> mgmtAgent.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-01T15:35:58.894Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.MgmtAgent resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.RKxD35P2Nz/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-8mw8v43n/pipeline@sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b to arohcpsvcdev.azurecr.io/ci-op-8mw8v43n/pipeline:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b.\nThe image will still be available under it's original digest sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b in the target registry.\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-8mw8v43n/pipeline@sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b arohcpsvcdev.azurecr.io/ci-op-8mw8v43n/pipeline:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b --from-registry-config /tmp/tmp.RKxD35P2Nz/containers/auth.json --to-registry-config /tmp/tmp.RKxD35P2Nz/containers/auth.json\n exit status 1"

time=2026-10-01T15:33:44.701Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> fleet.image.registry <nil>}:{<nil> fleet.image.repository <nil>}@{<nil> fleet.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-01T15:35:32.216Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.WQPJJsgkPI/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-49jbb95d/pipeline@sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 to arohcpsvcdev.azurecr.io/ci-op-49jbb95d/pipeline:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14.\nThe image will still be available under it's original digest sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 in the target registry.\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-49jbb95d/pipeline@sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 arohcpsvcdev.azurecr.io/ci-op-49jbb95d/pipeline:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 --from-registry-config /tmp/tmp.WQPJJsgkPI/containers/auth.json --to-registry-config /tmp/tmp.WQPJJsgkPI/containers/auth.json\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-z5kcbk" for cluster "cluster-candidate-4-20-z5kcbk"
Unexpected error:
    <*fmt.wrapError | 0xc00108f560>: 
    failed to create NodePool np-z5kcbk: failed waiting for nodepool="np-z5kcbk" for cluster "cluster-candidate-4-20-z5kcbk" in resourcegroup="rg-candidate-4-20-ffsrzblhv7vw" to finish creating: GET https://rp.j4881408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6f790e7-d2d5-4c8c-9960-78b89aeb4530
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6f790e7-d2d5-4c8c-9960-78b89aeb4530",
      "name": "b6f790e7-d2d5-4c8c-9960-78b89aeb4530",
      "status": "Failed",
      "startTime": "2026-10-01T15:10:35.278383091Z",
      "endTime": "2026-10-01T15:29:43.953476282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ffsrzblhv7vw-candidate-z5kcbk--managed/providers/Microsoft.Compute/virtualMachines/p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ffsrzblhv7vw-candidate-z5kcbk--managed/providers/Microsoft.Compute/virtualMachines/p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-z5kcbk" for cluster "cluster-candidate-4-20-z5kcbk"
Unexpected error:
    <*fmt.wrapError | 0xc00108f560>: 
    failed to create NodePool np-z5kcbk: failed waiting for nodepool="np-z5kcbk" for cluster "cluster-candidate-4-20-z5kcbk" in resourcegroup="rg-candidate-4-20-ffsrzblhv7vw" to finish creating: GET https://rp.j4881408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6f790e7-d2d5-4c8c-9960-78b89aeb4530
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6f790e7-d2d5-4c8c-9960-78b89aeb4530",
      "name": "b6f790e7-d2d5-4c8c-9960-78b89aeb4530",
      "status": "Failed",
      "startTime": "2026-10-01T15:10:35.278383091Z",
      "endTime": "2026-10-01T15:29:43.953476282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ffsrzblhv7vw-candidate-z5kcbk--managed/providers/Microsoft.Compute/virtualMachines/p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ffsrzblhv7vw-candidate-z5kcbk--managed/providers/Microsoft.Compute/virtualMachines/p9n7z6x8o1g2s9g-np-z5kcbk-fg5fk-94gqt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.40
Unexpected error:
    <*fmt.wrapErrors | 0xc0013de390>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-vvt8wh" resourcegroup="rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-vvt8wh in resource group rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.40
Unexpected error:
    <*fmt.wrapErrors | 0xc0013de390>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-vvt8wh" resourcegroup="rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-vvt8wh in resource group rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv, error: context deadline exceeded
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T15:34:25Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4147456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.82:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4147456/providers/microsoft.monitor/accounts/services-j4147456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-9976f6db4-kdw9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4147456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T15:35:24Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4147456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.82:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4147456/providers/microsoft.monitor/accounts/services-j4147456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-9976f6db4-kdw9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4147456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T14:26:18Z
  Ended:       2026-10-01T14:44:16Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4147456-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000ee8280>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-khlqvt2zwpg8" to finish creating: GET https://rp.j5360512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d2ae6a6-d320-4323-8352-4f27acc95ee3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d2ae6a6-d320-4323-8352-4f27acc95ee3",
      "name": "9d2ae6a6-d320-4323-8352-4f27acc95ee3",
      "status": "Failed",
      "startTime": "2026-10-01T14:13:06.877369112Z",
      "endTime": "2026-10-01T14:32:08.329973079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20260901-np-1-h5n6z-dgr5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-dgr5h to exist\nMachine v20260901-np-1-h5n6z-zgxpx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-zgxpx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-h5n6z-dgr5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-dgr5h to exist\nMachine v20260901-np-1-h5n6z-zgxpx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-zgxpx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000ee8280>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-khlqvt2zwpg8" to finish creating: GET https://rp.j5360512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d2ae6a6-d320-4323-8352-4f27acc95ee3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9d2ae6a6-d320-4323-8352-4f27acc95ee3",
      "name": "9d2ae6a6-d320-4323-8352-4f27acc95ee3",
      "status": "Failed",
      "startTime": "2026-10-01T14:13:06.877369112Z",
      "endTime": "2026-10-01T14:32:08.329973079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20260901-np-1-h5n6z-dgr5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-dgr5h to exist\nMachine v20260901-np-1-h5n6z-zgxpx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-zgxpx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-h5n6z-dgr5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-dgr5h to exist\nMachine v20260901-np-1-h5n6z-zgxpx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-khlqvt2zwpg8--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-h5n6z-zgxpx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000739be0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-v5pj4qqth2gl" to finish creating: GET https://rp.j3797248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca",
      "name": "523b7cc8-b248-458d-8692-3a89ce945aca",
      "status": "Failed",
      "startTime": "2026-10-01T13:50:45.474668418Z",
      "endTime": "2026-10-01T14:09:46.073738279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000739be0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-v5pj4qqth2gl" to finish creating: GET https://rp.j3797248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca",
      "name": "523b7cc8-b248-458d-8692-3a89ce945aca",
      "status": "Failed",
      "startTime": "2026-10-01T13:50:45.474668418Z",
      "endTime": "2026-10-01T14:09:46.073738279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T14:24:02Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5027584-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5027584/providers/microsoft.monitor/accounts/services-j5027584", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-65fddd48b7-ckqvd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5027584-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T14:24:02Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5027584-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5027584/providers/microsoft.monitor/accounts/services-j5027584", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-65fddd48b7-ckqvd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5027584-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f86500>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2pglwc5rj5zf" to finish creating: GET https://rp.j7109632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a4275a5-b602-4a40-b12f-be00bd2bbb9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a4275a5-b602-4a40-b12f-be00bd2bbb9c",
      "name": "7a4275a5-b602-4a40-b12f-be00bd2bbb9c",
      "status": "Failed",
      "startTime": "2026-10-01T12:32:10.987666018Z",
      "endTime": "2026-10-01T12:51:10.682185898Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j to exist\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j to exist\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f86500>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-2pglwc5rj5zf" to finish creating: GET https://rp.j7109632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a4275a5-b602-4a40-b12f-be00bd2bbb9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7a4275a5-b602-4a40-b12f-be00bd2bbb9c",
      "name": "7a4275a5-b602-4a40-b12f-be00bd2bbb9c",
      "status": "Failed",
      "startTime": "2026-10-01T12:32:10.987666018Z",
      "endTime": "2026-10-01T12:51:10.682185898Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j to exist\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-kqc7j to exist\nMachine j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-2pglwc5rj5zf--managed/providers/Microsoft.Compute/virtualMachines/j2v1i8x3h7v6d4g-np-1-62wr9-qt5zw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-jsgjhmrsczb4/cluster-candidate-4-23-tqz4r8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011a41c0>: 
    failed to create HCP cluster cluster-candidate-4-23-tqz4r8: failed waiting for cluster="cluster-candidate-4-23-tqz4r8" in resourcegroup="rg-candidate-4-23-jsgjhmrsczb4" to finish creating: GET https://rp.j9824640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "name": "3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "status": "Failed",
      "startTime": "2026-10-01T11:58:08.343305291Z",
      "endTime": "2026-10-01T12:27:12.911892104Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-jsgjhmrsczb4/cluster-candidate-4-23-tqz4r8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011a41c0>: 
    failed to create HCP cluster cluster-candidate-4-23-tqz4r8: failed waiting for cluster="cluster-candidate-4-23-tqz4r8" in resourcegroup="rg-candidate-4-23-jsgjhmrsczb4" to finish creating: GET https://rp.j9824640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "name": "3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "status": "Failed",
      "startTime": "2026-10-01T11:58:08.343305291Z",
      "endTime": "2026-10-01T12:27:12.911892104Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:20:58Z
  Ended:       2026-10-01T12:50:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f591f195-5e0c-485f-a44a-f0bfd5b23bc7"
  Description: Pod ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T12:20:58Z
  Ended:       2026-10-01T12:49:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f591f195-5e0c-485f-a44a-f0bfd5b23bc7"
  Description: Pod ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:32:56Z
  Ended:       2026-10-01T12:47:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T12:32:56Z
  Ended:       2026-10-01T12:46:56Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:19:00Z
  Ended:       2026-10-01T12:45:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T12:19:00Z
  Ended:       2026-10-01T12:45:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd has not matched the expected number of replicas for longer than 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013aaac8>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013aaac8>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002898f60>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-mbclvmsdh2bf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-mbclvmsdh2bf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002898f60>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-mbclvmsdh2bf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-mbclvmsdh2bf, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000a1ce80>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-6thznq" in resourcegroup="rg-np-version-upgrade-6thznq-8jq6fzpd5p6s" to finish creating: GET https://rp.j8115712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a86c7d1-5992-48c5-b0ad-533c78542d05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a86c7d1-5992-48c5-b0ad-533c78542d05",
      "name": "2a86c7d1-5992-48c5-b0ad-533c78542d05",
      "status": "Failed",
      "startTime": "2026-10-01T12:08:38.714435209Z",
      "endTime": "2026-10-01T12:27:38.568880925Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6thznq-8jq6fzpd5p6s-np-upgrade-6t-eb6955ce/providers/Microsoft.Compute/virtualMachines/y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6thznq-8jq6fzpd5p6s-np-upgrade-6t-eb6955ce/providers/Microsoft.Compute/virtualMachines/y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000a1ce80>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-6thznq" in resourcegroup="rg-np-version-upgrade-6thznq-8jq6fzpd5p6s" to finish creating: GET https://rp.j8115712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a86c7d1-5992-48c5-b0ad-533c78542d05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2a86c7d1-5992-48c5-b0ad-533c78542d05",
      "name": "2a86c7d1-5992-48c5-b0ad-533c78542d05",
      "status": "Failed",
      "startTime": "2026-10-01T12:08:38.714435209Z",
      "endTime": "2026-10-01T12:27:38.568880925Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6thznq-8jq6fzpd5p6s-np-upgrade-6t-eb6955ce/providers/Microsoft.Compute/virtualMachines/y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6thznq-8jq6fzpd5p6s-np-upgrade-6t-eb6955ce/providers/Microsoft.Compute/virtualMachines/y2f0v8q2i1x7o4s-npupgrade-4-20-q4knj-wkhj5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d27620>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-7vcvp4" in resourcegroup="rg-np-version-upgrade-7vcvp4-rc4bt9tf6jng" to finish creating: GET https://rp.j8197888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c76f634-2671-426d-a3ee-f0104d47c88f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c76f634-2671-426d-a3ee-f0104d47c88f",
      "name": "1c76f634-2671-426d-a3ee-f0104d47c88f",
      "status": "Failed",
      "startTime": "2026-10-01T12:11:01.569704477Z",
      "endTime": "2026-10-01T12:30:05.177016517Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7vcvp4-rc4bt9tf6jng-np-upgrade-7v-58e27b8c/providers/Microsoft.Compute/virtualMachines/p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7vcvp4-rc4bt9tf6jng-np-upgrade-7v-58e27b8c/providers/Microsoft.Compute/virtualMachines/p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d27620>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-7vcvp4" in resourcegroup="rg-np-version-upgrade-7vcvp4-rc4bt9tf6jng" to finish creating: GET https://rp.j8197888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c76f634-2671-426d-a3ee-f0104d47c88f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c76f634-2671-426d-a3ee-f0104d47c88f",
      "name": "1c76f634-2671-426d-a3ee-f0104d47c88f",
      "status": "Failed",
      "startTime": "2026-10-01T12:11:01.569704477Z",
      "endTime": "2026-10-01T12:30:05.177016517Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7vcvp4-rc4bt9tf6jng-np-upgrade-7v-58e27b8c/providers/Microsoft.Compute/virtualMachines/p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7vcvp4-rc4bt9tf6jng-np-upgrade-7v-58e27b8c/providers/Microsoft.Compute/virtualMachines/p9o2o7t3v3a5j3r-npupgrade-4-21-z7pzq-7bcjq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:11:06Z
  Ended:       2026-10-01T12:17:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8197888-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5vg8dltnlcbk8c17tn7tb3mpjhgjq9-u5u0k2w2t5m6a1t", pod="openshift-apiserver-58f5558dc-lmkqq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5vg8dltnlcbk8c17tn7tb3mpjhgjq9-u5u0k2w2t5m6a1t/openshift-apiserver-58f5558dc-lmkqq has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0009ae340>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7spzqz6xm5dj" to finish creating: GET https://rp.j8313344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a344fc73-234f-4877-890c-fd727d0da9ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a344fc73-234f-4877-890c-fd727d0da9ad",
      "name": "a344fc73-234f-4877-890c-fd727d0da9ad",
      "status": "Failed",
      "startTime": "2026-10-01T10:53:20.721727588Z",
      "endTime": "2026-10-01T11:22:26.523895596Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0009ae340>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7spzqz6xm5dj" to finish creating: GET https://rp.j8313344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a344fc73-234f-4877-890c-fd727d0da9ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a344fc73-234f-4877-890c-fd727d0da9ad",
      "name": "a344fc73-234f-4877-890c-fd727d0da9ad",
      "status": "Failed",
      "startTime": "2026-10-01T10:53:20.721727588Z",
      "endTime": "2026-10-01T11:22:26.523895596Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T11:13:31Z
  Ended:       2026-10-01T11:40:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8313344-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.63:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8313344/providers/microsoft.monitor/accounts/hcps-j8313344", namespace="ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-szg8z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T11:13:31Z
  Ended:       2026-10-01T11:40:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8313344-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.63:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8313344/providers/microsoft.monitor/accounts/hcps-j8313344", namespace="ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-szg8z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T11:27:28Z
  Ended:       2026-10-01T11:42:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8313344-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.63:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8313344/providers/microsoft.monitor/accounts/hcps-j8313344", namespace="ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-szg8z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T11:27:28Z
  Ended:       2026-10-01T11:42:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8313344-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.63:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8313344/providers/microsoft.monitor/accounts/hcps-j8313344", namespace="ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-szg8z", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T11:09:02Z
  Ended:       2026-10-01T11:15:01Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8313344-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T11:15:26Z
  Ended:       2026-10-01T11:45:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8313344-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.63:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8313344/providers/microsoft.monitor/accounts/hcps-j8313344", namespace="ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e873af76-f25c-41d3-aa91-50cf9a2d0d6d"
  Description: Pod ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T11:15:26Z
  Ended:       2026-10-01T11:45:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8313344-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.63:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8313344/providers/microsoft.monitor/accounts/hcps-j8313344", namespace="ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e873af76-f25c-41d3-aa91-50cf9a2d0d6d"
  Description: Pod ocm-arohcpci01-2t5ur8h7o9o78hglv40eq2k06fguq7dr-h7a4r0h7w1r6q5x/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kjl6sd" for cluster "cluster-candidate-4-20-kjl6sd"
Unexpected error:
    <*fmt.wrapError | 0xc000becf80>: 
    failed to create NodePool np-kjl6sd: failed waiting for nodepool="np-kjl6sd" for cluster "cluster-candidate-4-20-kjl6sd" in resourcegroup="rg-candidate-4-20-xmd65vxcpgjq" to finish creating: GET https://rp.j3793408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3d7a4d6-8c2c-4769-ada6-4519385de6eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3d7a4d6-8c2c-4769-ada6-4519385de6eb",
      "name": "e3d7a4d6-8c2c-4769-ada6-4519385de6eb",
      "status": "Failed",
      "startTime": "2026-10-01T10:58:58.306608832Z",
      "endTime": "2026-10-01T11:18:05.583630767Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmd65vxcpgjq-candidate-kjl6sd--managed/providers/Microsoft.Compute/virtualMachines/g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmd65vxcpgjq-candidate-kjl6sd--managed/providers/Microsoft.Compute/virtualMachines/g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kjl6sd" for cluster "cluster-candidate-4-20-kjl6sd"
Unexpected error:
    <*fmt.wrapError | 0xc000becf80>: 
    failed to create NodePool np-kjl6sd: failed waiting for nodepool="np-kjl6sd" for cluster "cluster-candidate-4-20-kjl6sd" in resourcegroup="rg-candidate-4-20-xmd65vxcpgjq" to finish creating: GET https://rp.j3793408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3d7a4d6-8c2c-4769-ada6-4519385de6eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3d7a4d6-8c2c-4769-ada6-4519385de6eb",
      "name": "e3d7a4d6-8c2c-4769-ada6-4519385de6eb",
      "status": "Failed",
      "startTime": "2026-10-01T10:58:58.306608832Z",
      "endTime": "2026-10-01T11:18:05.583630767Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmd65vxcpgjq-candidate-kjl6sd--managed/providers/Microsoft.Compute/virtualMachines/g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmd65vxcpgjq-candidate-kjl6sd--managed/providers/Microsoft.Compute/virtualMachines/g9v8a1r7h6n3u7o-np-kjl6sd-9kgtk-td2wq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-tqsp5m"
Unexpected error:
    <*fmt.wrapError | 0xc0016712e0>: 
    DNS for route host agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-tqsp5m"
Unexpected error:
    <*fmt.wrapError | 0xc0016712e0>: 
    DNS for route host agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cf3140>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-ll8zkb" in resourcegroup="rg-np-noedge-ll8zkb-7rrctv2mb529" to finish creating: GET https://rp.j3793408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48a26bba-177e-4cd1-987e-1d30da2cb320
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48a26bba-177e-4cd1-987e-1d30da2cb320",
      "name": "48a26bba-177e-4cd1-987e-1d30da2cb320",
      "status": "Failed",
      "startTime": "2026-10-01T11:20:22.361094887Z",
      "endTime": "2026-10-01T11:39:25.532208783Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ll8zkb-7rrctv2mb529-np-ne-ll8zkb--managed/providers/Microsoft.Compute/virtualMachines/s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ll8zkb-7rrctv2mb529-np-ne-ll8zkb--managed/providers/Microsoft.Compute/virtualMachines/s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cf3140>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-ll8zkb" in resourcegroup="rg-np-noedge-ll8zkb-7rrctv2mb529" to finish creating: GET https://rp.j3793408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48a26bba-177e-4cd1-987e-1d30da2cb320
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48a26bba-177e-4cd1-987e-1d30da2cb320",
      "name": "48a26bba-177e-4cd1-987e-1d30da2cb320",
      "status": "Failed",
      "startTime": "2026-10-01T11:20:22.361094887Z",
      "endTime": "2026-10-01T11:39:25.532208783Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ll8zkb-7rrctv2mb529-np-ne-ll8zkb--managed/providers/Microsoft.Compute/virtualMachines/s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-ll8zkb-7rrctv2mb529-np-ne-ll8zkb--managed/providers/Microsoft.Compute/virtualMachines/s9a5x6k1p7k1e0b-npnoedge-4-20-gngpw-mxx2j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000b3ce80>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-5km86c9kvr4n" to finish creating: GET https://rp.j6603520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b12f4581-fbe6-41af-91fe-4fb535b76c31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b12f4581-fbe6-41af-91fe-4fb535b76c31",
      "name": "b12f4581-fbe6-41af-91fe-4fb535b76c31",
      "status": "Failed",
      "startTime": "2026-10-01T10:38:29.13145534Z",
      "endTime": "2026-10-01T11:07:29.162156042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000b3ce80>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-5km86c9kvr4n" to finish creating: GET https://rp.j6603520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b12f4581-fbe6-41af-91fe-4fb535b76c31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b12f4581-fbe6-41af-91fe-4fb535b76c31",
      "name": "b12f4581-fbe6-41af-91fe-4fb535b76c31",
      "status": "Failed",
      "startTime": "2026-10-01T10:38:29.13145534Z",
      "endTime": "2026-10-01T11:07:29.162156042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T11:00:57Z
  Ended:       2026-10-01T11:29:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6603520-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.231:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6603520/providers/microsoft.monitor/accounts/hcps-j6603520", namespace="ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1b41462f-5134-4ae1-84b6-267321452766"
  Description: Pod ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T11:00:57Z
  Ended:       2026-10-01T11:29:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6603520-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.231:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6603520/providers/microsoft.monitor/accounts/hcps-j6603520", namespace="ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1b41462f-5134-4ae1-84b6-267321452766"
  Description: Pod ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T11:12:58Z
  Ended:       2026-10-01T11:26:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6603520-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.231:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6603520/providers/microsoft.monitor/accounts/hcps-j6603520", namespace="ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dlqbg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T11:12:58Z
  Ended:       2026-10-01T11:26:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6603520-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.231:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6603520/providers/microsoft.monitor/accounts/hcps-j6603520", namespace="ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dlqbg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T10:58:01Z
  Ended:       2026-10-01T11:24:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6603520-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.231:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6603520/providers/microsoft.monitor/accounts/hcps-j6603520", namespace="ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dlqbg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T10:58:01Z
  Ended:       2026-10-01T11:24:57Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6603520-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.231:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6603520/providers/microsoft.monitor/accounts/hcps-j6603520", namespace="ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dlqbg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5uk9glj077q5bi9aln4hj1tp2vq6b6-v20261001/etcd has not matched the expected number of replicas for longer than 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011e2de0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zccqlzzv6qtt" to finish creating: GET https://rp.j9592192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae7705c4-70ef-44f0-9615-eab5e5f02169
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae7705c4-70ef-44f0-9615-eab5e5f02169",
      "name": "ae7705c4-70ef-44f0-9615-eab5e5f02169",
      "status": "Failed",
      "startTime": "2026-10-01T10:47:02.754657732Z",
      "endTime": "2026-10-01T11:06:06.368173759Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zccqlzzv6qtt--managed/providers/Microsoft.Compute/virtualMachines/r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zccqlzzv6qtt--managed/providers/Microsoft.Compute/virtualMachines/r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011e2de0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zccqlzzv6qtt" to finish creating: GET https://rp.j9592192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae7705c4-70ef-44f0-9615-eab5e5f02169
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ae7705c4-70ef-44f0-9615-eab5e5f02169",
      "name": "ae7705c4-70ef-44f0-9615-eab5e5f02169",
      "status": "Failed",
      "startTime": "2026-10-01T10:47:02.754657732Z",
      "endTime": "2026-10-01T11:06:06.368173759Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zccqlzzv6qtt--managed/providers/Microsoft.Compute/virtualMachines/r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-zccqlzzv6qtt--managed/providers/Microsoft.Compute/virtualMachines/r1e6i6d7x6t4h4l-np-1-kwn7v-rwbjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-7lwzdj"
Unexpected error:
    <*fmt.wrapError | 0xc000d4db60>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-7lwzdj" in resourcegroup="rg-cp-ystream-deny-pods-4-21-f57xtv5kj7kt" to finish creating: GET https://rp.j0646784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/373f8d96-3295-4a4e-99cd-e1ca61e2d3b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/373f8d96-3295-4a4e-99cd-e1ca61e2d3b8",
      "name": "373f8d96-3295-4a4e-99cd-e1ca61e2d3b8",
      "status": "Failed",
      "startTime": "2026-10-01T10:42:02.427848987Z",
      "endTime": "2026-10-01T11:01:06.260697205Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-f57xtv5kj7kt-cp-ystream-de-07ffa0f0/providers/Microsoft.Compute/virtualMachines/g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-f57xtv5kj7kt-cp-ystream-de-07ffa0f0/providers/Microsoft.Compute/virtualMachines/g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-7lwzdj"
Unexpected error:
    <*fmt.wrapError | 0xc000d4db60>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-7lwzdj" in resourcegroup="rg-cp-ystream-deny-pods-4-21-f57xtv5kj7kt" to finish creating: GET https://rp.j0646784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/373f8d96-3295-4a4e-99cd-e1ca61e2d3b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/373f8d96-3295-4a4e-99cd-e1ca61e2d3b8",
      "name": "373f8d96-3295-4a4e-99cd-e1ca61e2d3b8",
      "status": "Failed",
      "startTime": "2026-10-01T10:42:02.427848987Z",
      "endTime": "2026-10-01T11:01:06.260697205Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-f57xtv5kj7kt-cp-ystream-de-07ffa0f0/providers/Microsoft.Compute/virtualMachines/g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-f57xtv5kj7kt-cp-ystream-de-07ffa0f0/providers/Microsoft.Compute/virtualMachines/g9q1i6d0k6r7g7r-np-deny-pods-62r5x-sc9r7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc00106e120>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-77mwfh658bt5" to finish creating: GET https://rp.j0646784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9393dda9-d86f-425b-b2fc-8763ba282289
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9393dda9-d86f-425b-b2fc-8763ba282289",
      "name": "9393dda9-d86f-425b-b2fc-8763ba282289",
      "status": "Failed",
      "startTime": "2026-10-01T10:43:10.823828182Z",
      "endTime": "2026-10-01T11:02:15.8524758Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-77mwfh658bt5--managed/providers/Microsoft.Compute/virtualMachines/h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-77mwfh658bt5--managed/providers/Microsoft.Compute/virtualMachines/h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc00106e120>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-77mwfh658bt5" to finish creating: GET https://rp.j0646784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9393dda9-d86f-425b-b2fc-8763ba282289
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9393dda9-d86f-425b-b2fc-8763ba282289",
      "name": "9393dda9-d86f-425b-b2fc-8763ba282289",
      "status": "Failed",
      "startTime": "2026-10-01T10:43:10.823828182Z",
      "endTime": "2026-10-01T11:02:15.8524758Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-77mwfh658bt5--managed/providers/Microsoft.Compute/virtualMachines/h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-77mwfh658bt5--managed/providers/Microsoft.Compute/virtualMachines/h9o6s5u4b5f1f2v-np-1-9bfq2-x8ptp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T10:50:49Z
  Ended:       2026-10-01T11:10:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0646784-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5ugl1cv1c2arhvq63uatug83p2l5bt-c7e4k0r5n7w9n8r", pod="ignition-server-59497d8747-7tdhl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5ugl1cv1c2arhvq63uatug83p2l5bt-c7e4k0r5n7w9n8r/ignition-server-59497d8747-7tdhl has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000c8f000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-79j6gx" in resourcegroup="rg-np-noedge-79j6gx-2qlfj7fcvd5l" to finish creating: GET https://rp.j0573696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d982722-8bc8-4822-b239-6489bf62ebbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d982722-8bc8-4822-b239-6489bf62ebbe",
      "name": "8d982722-8bc8-4822-b239-6489bf62ebbe",
      "status": "Failed",
      "startTime": "2026-10-01T09:40:31.718149787Z",
      "endTime": "2026-10-01T09:59:33.513734855Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-79j6gx-2qlfj7fcvd5l-np-ne-79j6gx--managed/providers/Microsoft.Compute/virtualMachines/o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-79j6gx-2qlfj7fcvd5l-np-ne-79j6gx--managed/providers/Microsoft.Compute/virtualMachines/o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000c8f000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-79j6gx" in resourcegroup="rg-np-noedge-79j6gx-2qlfj7fcvd5l" to finish creating: GET https://rp.j0573696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d982722-8bc8-4822-b239-6489bf62ebbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d982722-8bc8-4822-b239-6489bf62ebbe",
      "name": "8d982722-8bc8-4822-b239-6489bf62ebbe",
      "status": "Failed",
      "startTime": "2026-10-01T09:40:31.718149787Z",
      "endTime": "2026-10-01T09:59:33.513734855Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-79j6gx-2qlfj7fcvd5l-np-ne-79j6gx--managed/providers/Microsoft.Compute/virtualMachines/o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-79j6gx-2qlfj7fcvd5l-np-ne-79j6gx--managed/providers/Microsoft.Compute/virtualMachines/o5b8c2t8b4t6b5u-npnoedge-4-20-p6hg9-hbrgs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T07:22:46Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j1943424-mgmt-1", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1943424/providers/microsoft.monitor/accounts/services-j1943424", namespace="arobit", pod="arobit-forwarder-b8lv8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fe636689-1a0b-4733-829d-e26d9c7f983a"
  Description: Container fluentbit in pod arobit/arobit-forwarder-b8lv8 on cluster ci01-j1943424-mgmt-1 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T07:22:47Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j1943424-mgmt-1", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1943424/providers/microsoft.monitor/accounts/services-j1943424", namespace="arobit", pod="arobit-forwarder-b8lv8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fe636689-1a0b-4733-829d-e26d9c7f983a"
  Description: Container fluentbit in pod arobit/arobit-forwarder-b8lv8 on cluster ci01-j1943424-mgmt-1 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T07:46:18Z
  Ended:       2026-10-01T07:52:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1943424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5rfje5rojgjp94o6gjnikcjkjfl6fn-y1j1s4w4k7h3t0i", pod="ingress-operator-c7575bbcd-p4vgt", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5rfje5rojgjp94o6gjnikcjkjfl6fn-y1j1s4w4k7h3t0i/ingress-operator-c7575bbcd-p4vgt has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T02:42:44Z
  Ended:       2026-10-01T02:58:43Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1433728-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:205]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc0014b5300>: 
    node pool upgrade verification failed: h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); nodes=[{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":30},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:205]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc0014b5300>: 
    node pool upgrade verification failed: h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); nodes=[{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":30},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47}]
    ...

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T02:22:58Z
  Ended:       2026-10-01T02:36:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2039552-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T02:29:53Z
  Ended:       2026-10-01T02:38:53Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2039552-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T02:18:14Z
  Ended:       2026-10-01T02:24:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2924800-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5msf76kovdd4mdu19ofligrvegml9k-ea-list8lfh58", pod="kube-apiserver-9974ddf74-9mlxr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5msf76kovdd4mdu19ofligrvegml9k-ea-list8lfh58/kube-apiserver-9974ddf74-9mlxr has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk"
Unexpected error:
    <*fmt.wrapError | 0xc000941160>: 
    failed to create NodePool np-jv9hvk: failed waiting for nodepool="np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk" in resourcegroup="rg-candidate-4-20-6nfp6mj4gt68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "name": "29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "status": "Failed",
      "startTime": "2026-10-01T20:46:24.926645724Z",
      "endTime": "2026-10-01T21:05:28.967720321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk"
Unexpected error:
    <*fmt.wrapError | 0xc000941160>: 
    failed to create NodePool np-jv9hvk: failed waiting for nodepool="np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk" in resourcegroup="rg-candidate-4-20-6nfp6mj4gt68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "name": "29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "status": "Failed",
      "startTime": "2026-10-01T20:46:24.926645724Z",
      "endTime": "2026-10-01T21:05:28.967720321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012c9e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fxmrv96q875w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269",
      "name": "a1048b90-244a-4c5a-ba89-e1135a68a269",
      "status": "Failed",
      "startTime": "2026-10-01T20:23:38.274618399Z",
      "endTime": "2026-10-01T20:42:38.037012934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012c9e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fxmrv96q875w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269",
      "name": "a1048b90-244a-4c5a-ba89-e1135a68a269",
      "status": "Failed",
      "startTime": "2026-10-01T20:23:38.274618399Z",
      "endTime": "2026-10-01T20:42:38.037012934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-282pdp" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000fe7f50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-qszf5tc8smsd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-282pdp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-282pdp" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000fe7f50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-qszf5tc8smsd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-282pdp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-fpkhgg" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0013695c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-x2qm64ntnf4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-fpkhgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-fpkhgg" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0013695c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-x2qm64ntnf4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-fpkhgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-zhcqgm" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000f2be90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-bkxc6qpdr2kc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-zhcqgm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-zhcqgm" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000f2be90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-bkxc6qpdr2kc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-zhcqgm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-ncq7tv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0014ac6f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jnrgf69jtww7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-ncq7tv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-ncq7tv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0014ac6f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jnrgf69jtww7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-ncq7tv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5jgcvl" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00156e510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-qqh9mh2dgv5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5jgcvl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5jgcvl" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00156e510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-qqh9mh2dgv5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5jgcvl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-cznf64" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0018c4c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-z6bx6clhw9nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-cznf64
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-cznf64" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0018c4c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-z6bx6clhw9nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-cznf64
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5lppxg" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c77c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-9qkw2hc8j64k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5lppxg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5lppxg" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c77c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-9qkw2hc8j64k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5lppxg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-rtvfwf" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0012da990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zzffnqvsn5bk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-rtvfwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-rtvfwf" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0012da990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zzffnqvsn5bk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-rtvfwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-59lhdf" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b51650>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-zncx9w9pdbrq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-59lhdf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-59lhdf" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b51650>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-zncx9w9pdbrq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-59lhdf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-5gft95" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0015d78c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-8c7gqffnfmhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-5gft95
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-5gft95" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0015d78c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-8c7gqffnfmhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-5gft95
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-tvg8ls" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001483050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-8ptjnkq2z7df/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-tvg8ls
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-tvg8ls" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001483050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-8ptjnkq2z7df/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-tvg8ls
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fbsgr5" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00153e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rfwzkqxm7bss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-fbsgr5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fbsgr5" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00153e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rfwzkqxm7bss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-fbsgr5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-gfc5ks" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b4b2f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2k4s9c5c5gt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-gfc5ks
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-gfc5ks" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b4b2f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2k4s9c5c5gt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-gfc5ks
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-skgz2q" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0014816b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-j4gwfrfwllb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-skgz2q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-skgz2q" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0014816b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-j4gwfrfwllb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-skgz2q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-97ngnf" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0009a0ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-l7xhqzrtl9sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-97ngnf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-97ngnf" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0009a0ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-l7xhqzrtl9sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-97ngnf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-84kjqn" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00159a510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-9tsgcnv7kf56/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-84kjqn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-84kjqn" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00159a510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-9tsgcnv7kf56/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-84kjqn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-6vqlpz" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001492180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-6954mm5k46dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-6vqlpz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-6vqlpz" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001492180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-6954mm5k46dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-6vqlpz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-gkw8ql" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0011fba70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-lzvgwqsshhgh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-gkw8ql
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-gkw8ql" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0011fba70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-lzvgwqsshhgh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-gkw8ql
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qkh6np" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000af7500>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-jqv2lkbmlggl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qkh6np
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qkh6np" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000af7500>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-jqv2lkbmlggl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qkh6np
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5xch77" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001152bd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-869wg248knm4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5xch77
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5xch77" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001152bd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-869wg248knm4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5xch77
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-rngq5s" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000ccdb30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-mlc5mpzcvj8q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rngq5s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-rngq5s" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000ccdb30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-mlc5mpzcvj8q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rngq5s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-zx7mxd" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0018b1e90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gvghfmdnj9kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-zx7mxd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-zx7mxd" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0018b1e90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gvghfmdnj9kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-zx7mxd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-tkvcqf" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00155e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-9l9l89pgr4m5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-tkvcqf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-tkvcqf" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00155e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-9l9l89pgr4m5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-tkvcqf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-4p7wkl" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc001587980>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2dqnlv5bkbch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-4p7wkl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-4p7wkl" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc001587980>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2dqnlv5bkbch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-4p7wkl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00109d0c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s7p6hptqhbvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1",
      "name": "357a31b9-a87e-4b62-a8b5-196b066234f1",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:38.669148771Z",
      "endTime": "2026-10-01T03:17:44.618360897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00109d0c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s7p6hptqhbvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1",
      "name": "357a31b9-a87e-4b62-a8b5-196b066234f1",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:38.669148771Z",
      "endTime": "2026-10-01T03:17:44.618360897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-tsd828" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000ab0180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-qpv6d29gz4jl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-tsd828
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-tsd828" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000ab0180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-qpv6d29gz4jl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-tsd828
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-q5mklf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00118f7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-rcvrxff9wvxp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-q5mklf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-q5mklf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00118f7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-rcvrxff9wvxp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-q5mklf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-gswqqb" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0005f0a80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-nmmbsqvg94ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gswqqb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-gswqqb" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0005f0a80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-nmmbsqvg94ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gswqqb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-vnllsd" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0015a4150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-5lmngc5rw784/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-vnllsd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-vnllsd" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0015a4150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-5lmngc5rw784/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-vnllsd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00057dcc0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jspphct49kmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb",
      "name": "4a191606-d1f9-418e-b590-0307f5435adb",
      "status": "Failed",
      "startTime": "2026-10-01T01:58:13.545499376Z",
      "endTime": "2026-10-01T02:17:18.498756095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00057dcc0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jspphct49kmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb",
      "name": "4a191606-d1f9-418e-b590-0307f5435adb",
      "status": "Failed",
      "startTime": "2026-10-01T01:58:13.545499376Z",
      "endTime": "2026-10-01T02:17:18.498756095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-2crl7w" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0015c2c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-bdf2955zlsb2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2crl7w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-2crl7w" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0015c2c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-bdf2955zlsb2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2crl7w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4jz6tz" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000a1c2d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wjhccp827grc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4jz6tz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4jz6tz" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000a1c2d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wjhccp827grc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4jz6tz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000f4b7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-5rjr485pnvns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000f4b7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-5rjr485pnvns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001dfc720>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-72flmh48df4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001dfc720>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-72flmh48df4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-47sckvjgzdjp/cluster-nightly-4-22-wwdkq6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d874a0>: 
    failed to create HCP cluster cluster-nightly-4-22-wwdkq6: failed waiting for cluster="cluster-nightly-4-22-wwdkq6" in resourcegroup="rg-nightly-4-22-47sckvjgzdjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "name": "4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "status": "Failed",
      "startTime": "2026-10-01T23:33:29.768708196Z",
      "endTime": "2026-10-02T00:02:35.156751805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-47sckvjgzdjp/cluster-nightly-4-22-wwdkq6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d874a0>: 
    failed to create HCP cluster cluster-nightly-4-22-wwdkq6: failed waiting for cluster="cluster-nightly-4-22-wwdkq6" in resourcegroup="rg-nightly-4-22-47sckvjgzdjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "name": "4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "status": "Failed",
      "startTime": "2026-10-01T23:33:29.768708196Z",
      "endTime": "2026-10-02T00:02:35.156751805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b3d180>: 
    failed to create HCP cluster np-dg-minor-7t94dm: failed waiting for cluster="np-dg-minor-7t94dm" in resourcegroup="rg-np-dg-minor-7t94dm-kmpzpbhvdzdx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b",
      "name": "5b29ed00-0226-42af-9fb9-c6317a83dd8b",
      "status": "Failed",
      "startTime": "2026-10-02T00:05:37.505434862Z",
      "endTime": "2026-10-02T00:34:39.296827861Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b3d180>: 
    failed to create HCP cluster np-dg-minor-7t94dm: failed waiting for cluster="np-dg-minor-7t94dm" in resourcegroup="rg-np-dg-minor-7t94dm-kmpzpbhvdzdx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b29ed00-0226-42af-9fb9-c6317a83dd8b",
      "name": "5b29ed00-0226-42af-9fb9-c6317a83dd8b",
      "status": "Failed",
      "startTime": "2026-10-02T00:05:37.505434862Z",
      "endTime": "2026-10-02T00:34:39.296827861Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0013b7a10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-55xcrpmcpjd5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0013b7a10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-55xcrpmcpjd5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-szptjh" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000f34e40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-szptjh: failed waiting for cluster="cp-ystream-upgrade-4-23-szptjh" in resourcegroup="rg-cp-ystream-upgrade-4-23-46pm69d84hvp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "name": "780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "status": "Failed",
      "startTime": "2026-10-01T23:37:58.369447145Z",
      "endTime": "2026-10-02T00:06:59.154886423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-szptjh" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000f34e40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-szptjh: failed waiting for cluster="cp-ystream-upgrade-4-23-szptjh" in resourcegroup="rg-cp-ystream-upgrade-4-23-46pm69d84hvp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "name": "780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "status": "Failed",
      "startTime": "2026-10-01T23:37:58.369447145Z",
      "endTime": "2026-10-02T00:06:59.154886423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc000ed4500>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ngvxnx" in resourcegroup="rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ngvxnx/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc000ed4500>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ngvxnx" in resourcegroup="rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ngvxnx/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq"
Unexpected error:
    <*fmt.wrapError | 0xc0012f2520>: 
    failed to create NodePool np-ks75xq: failed starting nodepool creation "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" in resourcegroup="rg-nightly-4-20-2zfvw4gzpm77": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2zfvw4gzpm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-ks75xq/nodePools/np-ks75xq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq"
Unexpected error:
    <*fmt.wrapError | 0xc0012f2520>: 
    failed to create NodePool np-ks75xq: failed starting nodepool creation "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" in resourcegroup="rg-nightly-4-20-2zfvw4gzpm77": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2zfvw4gzpm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-ks75xq/nodePools/np-ks75xq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000a6f140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwrdblk56ldw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000a6f140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwrdblk56ldw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-5vfvkg" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc00092f060>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-5vfvkg: failed waiting for cluster="cp-ystream-upgrade-5-0-5vfvkg" in resourcegroup="rg-cp-ystream-upgrade-5-0-c88htx5v6jhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c",
      "name": "e8198907-58f9-403b-976d-4cda3e2ce98c",
      "status": "Failed",
      "startTime": "2026-10-01T23:38:13.440452696Z",
      "endTime": "2026-10-02T00:07:19.146440053Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-5vfvkg" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc00092f060>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-5vfvkg: failed waiting for cluster="cp-ystream-upgrade-5-0-5vfvkg" in resourcegroup="rg-cp-ystream-upgrade-5-0-c88htx5v6jhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8198907-58f9-403b-976d-4cda3e2ce98c",
      "name": "e8198907-58f9-403b-976d-4cda3e2ce98c",
      "status": "Failed",
      "startTime": "2026-10-01T23:38:13.440452696Z",
      "endTime": "2026-10-02T00:07:19.146440053Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014996b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-10-01-083859, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014996b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-10-01-083859, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-bjmvhq at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0010869e0>: 
    failed to create HCP cluster np-skip-minor-bjmvhq: failed waiting for cluster="np-skip-minor-bjmvhq" in resourcegroup="rg-np-skip-minor-bjmvhq-v7kckbmqz7jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3",
      "name": "94714044-9af3-4f81-b892-f8e47c0efbb3",
      "status": "Failed",
      "startTime": "2026-10-02T00:05:39.688922023Z",
      "endTime": "2026-10-02T00:34:39.296960872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-bjmvhq at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0010869e0>: 
    failed to create HCP cluster np-skip-minor-bjmvhq: failed waiting for cluster="np-skip-minor-bjmvhq" in resourcegroup="rg-np-skip-minor-bjmvhq-v7kckbmqz7jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3",
      "name": "94714044-9af3-4f81-b892-f8e47c0efbb3",
      "status": "Failed",
      "startTime": "2026-10-02T00:05:39.688922023Z",
      "endTime": "2026-10-02T00:34:39.296960872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc001138270>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-r822p4wshpfb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc001138270>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-r822p4wshpfb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001797968>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7fp2gpvrmpv5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc",
      "name": "fbace7f5-c615-4f66-9959-02331d194dcc",
      "status": "Failed",
      "startTime": "2026-10-01T23:17:05.438751852Z",
      "endTime": "2026-10-01T23:36:05.105086237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001797968>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7fp2gpvrmpv5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc",
      "name": "fbace7f5-c615-4f66-9959-02331d194dcc",
      "status": "Failed",
      "startTime": "2026-10-01T23:17:05.438751852Z",
      "endTime": "2026-10-01T23:36:05.105086237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0010769c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g5k9xrk5kbzf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680",
      "name": "1864d250-37c2-44c1-bbbf-426f923fe680",
      "status": "Failed",
      "startTime": "2026-10-01T16:25:59.009326346Z",
      "endTime": "2026-10-01T16:45:03.114189778Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0010769c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g5k9xrk5kbzf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680",
      "name": "1864d250-37c2-44c1-bbbf-426f923fe680",
      "status": "Failed",
      "startTime": "2026-10-01T16:25:59.009326346Z",
      "endTime": "2026-10-01T16:45:03.114189778Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cbe8e0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-f85th5" in resourcegroup="rg-np-noedge-f85th5-2lj4rn8xdzrh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "name": "3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "status": "Failed",
      "startTime": "2026-10-01T18:02:34.984361176Z",
      "endTime": "2026-10-01T18:21:38.6870989Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cbe8e0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-f85th5" in resourcegroup="rg-np-noedge-f85th5-2lj4rn8xdzrh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "name": "3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "status": "Failed",
      "startTime": "2026-10-01T18:02:34.984361176Z",
      "endTime": "2026-10-01T18:21:38.6870989Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001802258>: 
    resource group "rg-zstream-upgrade-4-20-26f5nw6rjl84": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-m4xqll" in resourcegroup="rg-zstream-upgrade-4-20-26f5nw6rjl84" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "name": "fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "status": "Failed",
      "startTime": "2026-10-01T17:28:43.347525555Z",
      "endTime": "2026-10-01T17:52:46.138429989Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t648pvv6ett9l7cgmh574bcs7i8oe4q still exists (deletion dispatched at 2026-10-01T17:28:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001802258>: 
    resource group "rg-zstream-upgrade-4-20-26f5nw6rjl84": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-m4xqll" in resourcegroup="rg-zstream-upgrade-4-20-26f5nw6rjl84" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "name": "fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "status": "Failed",
      "startTime": "2026-10-01T17:28:43.347525555Z",
      "endTime": "2026-10-01T17:52:46.138429989Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t648pvv6ett9l7cgmh574bcs7i8oe4q still exists (deletion dispatched at 2026-10-01T17:28:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001f80160>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001f80160>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000638600>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2nzmgkxwzltg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535",
      "name": "eaa1d5d5-fc6a-4816-a207-694d27453535",
      "status": "Failed",
      "startTime": "2026-10-01T05:17:35.937388036Z",
      "endTime": "2026-10-01T05:36:39.648087048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000638600>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2nzmgkxwzltg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535",
      "name": "eaa1d5d5-fc6a-4816-a207-694d27453535",
      "status": "Failed",
      "startTime": "2026-10-01T05:17:35.937388036Z",
      "endTime": "2026-10-01T05:36:39.648087048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw"
Unexpected error:
    <*fmt.wrapError | 0xc0005cc920>: 
    failed to create NodePool np-ttzgdw: failed waiting for nodepool="np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw" in resourcegroup="rg-candidate-4-23-2gh8rvt7nqts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e",
      "name": "77801f28-f3c8-4c43-a265-b1074104615e",
      "status": "Failed",
      "startTime": "2026-10-01T05:58:38.023066435Z",
      "endTime": "2026-10-01T06:17:40.99775088Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw"
Unexpected error:
    <*fmt.wrapError | 0xc0005cc920>: 
    failed to create NodePool np-ttzgdw: failed waiting for nodepool="np-ttzgdw" for cluster "cluster-candidate-4-23-ttzgdw" in resourcegroup="rg-candidate-4-23-2gh8rvt7nqts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/77801f28-f3c8-4c43-a265-b1074104615e",
      "name": "77801f28-f3c8-4c43-a265-b1074104615e",
      "status": "Failed",
      "startTime": "2026-10-01T05:58:38.023066435Z",
      "endTime": "2026-10-01T06:17:40.99775088Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-4g5n9 to exist\nMachine e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2gh8rvt7nqts-candidate-ttzgdw--managed/providers/Microsoft.Compute/virtualMachines/e0w1o8l4a5a0f1h-np-ttzgdw-2xppc-bzxx8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f94940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-shmswfdhjbs5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f",
      "name": "5bf51a3f-7e39-4185-a2e3-46167169303f",
      "status": "Failed",
      "startTime": "2026-10-01T05:34:42.041881998Z",
      "endTime": "2026-10-01T05:53:44.016651744Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f94940>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-shmswfdhjbs5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/5bf51a3f-7e39-4185-a2e3-46167169303f",
      "name": "5bf51a3f-7e39-4185-a2e3-46167169303f",
      "status": "Failed",
      "startTime": "2026-10-01T05:34:42.041881998Z",
      "endTime": "2026-10-01T05:53:44.016651744Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-shmswfdhjbs5--managed/providers/Microsoft.Compute/virtualMachines/l0l4y2g7i7s8y9y-np-1-lm9mg-8xxtg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc0012cb640>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc0012cb640>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc002d00360>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kssqgrfmgk4t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c",
      "name": "fef1a167-cbd5-4042-994e-c159a5e7bb3c",
      "status": "Failed",
      "startTime": "2026-10-01T05:33:21.995938468Z",
      "endTime": "2026-10-01T05:52:24.037057581Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc002d00360>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kssqgrfmgk4t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fef1a167-cbd5-4042-994e-c159a5e7bb3c",
      "name": "fef1a167-cbd5-4042-994e-c159a5e7bb3c",
      "status": "Failed",
      "startTime": "2026-10-01T05:33:21.995938468Z",
      "endTime": "2026-10-01T05:52:24.037057581Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-kssqgrfmgk4t--managed/providers/Microsoft.Compute/virtualMachines/d3e7v5o4i7z3m4f-np-1-sm8km-p4tbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-kbkkg2" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc00140fc80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-fw66sfpzcwgg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-kbkkg2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-kbkkg2" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc00140fc80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-fw66sfpzcwgg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-kbkkg2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xwvg9x" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000e80e40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-n4rqqc24skhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xwvg9x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xwvg9x" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000e80e40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-n4rqqc24skhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xwvg9x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7ktnhw" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00104ef60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-qqs544jdcqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7ktnhw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-7ktnhw" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00104ef60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-qqs544jdcqf6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-7ktnhw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-24sw5t" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000cdd5c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vhz5sh8l55g9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-24sw5t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-24sw5t" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000cdd5c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vhz5sh8l55g9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-24sw5t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-7wwtdz" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0012d2210>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-w7lfjqpj8rr7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-7wwtdz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-7wwtdz" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0012d2210>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-w7lfjqpj8rr7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-7wwtdz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-gdj89p" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00157dbf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rb86w8ww8sjw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gdj89p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-gdj89p" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00157dbf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rb86w8ww8sjw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gdj89p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4mwqz2" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00144a120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jlsd79zkk6hh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4mwqz2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4mwqz2" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00144a120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jlsd79zkk6hh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4mwqz2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-l9fj7m" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0015e41b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-4mvbdm9qd7qf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-l9fj7m
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-l9fj7m" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0015e41b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-4mvbdm9qd7qf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-l9fj7m
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-4fwf7q" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001226630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wnp2jmpdpwln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4fwf7q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-4fwf7q" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001226630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wnp2jmpdpwln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4fwf7q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-gvphp6" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0010f8390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qgd4bkmfx2lq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-gvphp6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-gvphp6" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0010f8390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qgd4bkmfx2lq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-gvphp6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd"
Unexpected error:
    <*fmt.wrapError | 0xc001012e40>: 
    failed to create NodePool np-6g6tzd: failed waiting for nodepool="np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd" in resourcegroup="rg-candidate-4-22-xz7x2977wmz2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "name": "3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:22.461339415Z",
      "endTime": "2026-10-01T23:41:27.247269633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: Creating: virtualmachine creating or updating\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd"
Unexpected error:
    <*fmt.wrapError | 0xc001012e40>: 
    failed to create NodePool np-6g6tzd: failed waiting for nodepool="np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd" in resourcegroup="rg-candidate-4-22-xz7x2977wmz2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "name": "3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:22.461339415Z",
      "endTime": "2026-10-01T23:41:27.247269633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: Creating: virtualmachine creating or updating\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f50a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-6svllmjvpbpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b",
      "name": "7c5e553a-2bca-431a-819c-94122ad78b6b",
      "status": "Failed",
      "startTime": "2026-10-02T00:04:50.703195637Z",
      "endTime": "2026-10-02T00:23:53.181881109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f50a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-6svllmjvpbpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b",
      "name": "7c5e553a-2bca-431a-819c-94122ad78b6b",
      "status": "Failed",
      "startTime": "2026-10-02T00:04:50.703195637Z",
      "endTime": "2026-10-02T00:23:53.181881109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz"
Unexpected error:
    <*fmt.wrapError | 0xc000cee380>: 
    failed to create NodePool np-58mkpz: failed waiting for nodepool="np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz" in resourcegroup="rg-candidate-4-21-8w4nf2bg8hwt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "name": "22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:00.007708534Z",
      "endTime": "2026-10-01T23:41:07.215796872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: Creating: virtualmachine creating or updating\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz"
Unexpected error:
    <*fmt.wrapError | 0xc000cee380>: 
    failed to create NodePool np-58mkpz: failed waiting for nodepool="np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz" in resourcegroup="rg-candidate-4-21-8w4nf2bg8hwt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "name": "22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:00.007708534Z",
      "endTime": "2026-10-01T23:41:07.215796872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: Creating: virtualmachine creating or updating\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv"
Unexpected error:
    <*fmt.wrapError | 0xc000dbc340>: 
    failed to create NodePool np-nz54xv: failed waiting for nodepool="np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv" in resourcegroup="rg-candidate-4-23-87g9stccjxzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "name": "7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:46.805553224Z",
      "endTime": "2026-10-01T23:41:47.237736739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: Creating: virtualmachine creating or updating\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv"
Unexpected error:
    <*fmt.wrapError | 0xc000dbc340>: 
    failed to create NodePool np-nz54xv: failed waiting for nodepool="np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv" in resourcegroup="rg-candidate-4-23-87g9stccjxzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "name": "7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:46.805553224Z",
      "endTime": "2026-10-01T23:41:47.237736739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: Creating: virtualmachine creating or updating\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000a47540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-rj7chpbn7z2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "name": "dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "status": "Failed",
      "startTime": "2026-10-01T22:58:07.666255851Z",
      "endTime": "2026-10-01T23:17:14.973053483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-9d89g-jglhd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-9d89g-jglhd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000a47540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-rj7chpbn7z2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "name": "dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "status": "Failed",
      "startTime": "2026-10-01T22:58:07.666255851Z",
      "endTime": "2026-10-01T23:17:14.973053483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-9d89g-jglhd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-9d89g-jglhd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz"
Unexpected error:
    <*fmt.wrapError | 0xc000acaa40>: 
    failed to create NodePool np-8gxplz: failed waiting for nodepool="np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz" in resourcegroup="rg-candidate-5-1-kvc5d4thszfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3",
      "name": "bde76db6-b7ac-422e-8660-ba8904affbd3",
      "status": "Failed",
      "startTime": "2026-10-01T23:26:31.681894722Z",
      "endTime": "2026-10-01T23:45:37.242801753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: Creating: virtualmachine creating or updating\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz"
Unexpected error:
    <*fmt.wrapError | 0xc000acaa40>: 
    failed to create NodePool np-8gxplz: failed waiting for nodepool="np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz" in resourcegroup="rg-candidate-5-1-kvc5d4thszfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3",
      "name": "bde76db6-b7ac-422e-8660-ba8904affbd3",
      "status": "Failed",
      "startTime": "2026-10-01T23:26:31.681894722Z",
      "endTime": "2026-10-01T23:45:37.242801753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: Creating: virtualmachine creating or updating\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019901c8>: 
    resource group "oidc-wi-nq9xg56z8ptc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nq9xg56z8ptc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "name": "b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "status": "Failed",
      "startTime": "2026-10-02T00:11:10.101784102Z",
      "endTime": "2026-10-02T00:35:13.452817702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t6a9r7piu3mrhnemk9adlcvlla5mb7v still exists (deletion dispatched at 2026-10-02T00:11:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019901c8>: 
    resource group "oidc-wi-nq9xg56z8ptc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nq9xg56z8ptc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "name": "b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "status": "Failed",
      "startTime": "2026-10-02T00:11:10.101784102Z",
      "endTime": "2026-10-02T00:35:13.452817702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t6a9r7piu3mrhnemk9adlcvlla5mb7v still exists (deletion dispatched at 2026-10-02T00:11:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d82400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-g79pk8vm4jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "name": "d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "status": "Failed",
      "startTime": "2026-10-01T23:53:42.212331844Z",
      "endTime": "2026-10-02T00:12:43.153935456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: Creating: virtualmachine creating or updating\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d82400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-g79pk8vm4jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "name": "d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "status": "Failed",
      "startTime": "2026-10-01T23:53:42.212331844Z",
      "endTime": "2026-10-02T00:12:43.153935456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: Creating: virtualmachine creating or updating\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f200c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4rdzlcbp8jqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931",
      "name": "7941e9fc-eabb-4912-b47f-1619c832d931",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:15.574287096Z",
      "endTime": "2026-10-01T23:21:15.149755975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f200c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4rdzlcbp8jqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931",
      "name": "7941e9fc-eabb-4912-b47f-1619c832d931",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:15.574287096Z",
      "endTime": "2026-10-01T23:21:15.149755975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k96n75" for cluster "cluster-candidate-5-0-k96n75"
Unexpected error:
    <*fmt.wrapError | 0xc000a16440>: 
    failed to create NodePool np-k96n75: failed waiting for nodepool="np-k96n75" for cluster "cluster-candidate-5-0-k96n75" in resourcegroup="rg-candidate-5-0-gclccxznszk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "name": "e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "status": "Failed",
      "startTime": "2026-10-01T23:23:36.842890739Z",
      "endTime": "2026-10-01T23:42:37.244899482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: Creating: virtualmachine creating or updating\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k96n75" for cluster "cluster-candidate-5-0-k96n75"
Unexpected error:
    <*fmt.wrapError | 0xc000a16440>: 
    failed to create NodePool np-k96n75: failed waiting for nodepool="np-k96n75" for cluster "cluster-candidate-5-0-k96n75" in resourcegroup="rg-candidate-5-0-gclccxznszk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "name": "e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "status": "Failed",
      "startTime": "2026-10-01T23:23:36.842890739Z",
      "endTime": "2026-10-01T23:42:37.244899482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: Creating: virtualmachine creating or updating\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001c16198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-mrv84pkzs8gp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "name": "c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:38.499943334Z",
      "endTime": "2026-10-01T23:21:45.00235976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001c16198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-mrv84pkzs8gp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "name": "c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:38.499943334Z",
      "endTime": "2026-10-01T23:21:45.00235976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv"
Unexpected error:
    <*fmt.wrapError | 0xc001076a40>: 
    failed to create NodePool np-bl9ftv: failed waiting for nodepool="np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv" in resourcegroup="rg-candidate-4-20-x4gs2n6gstps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "name": "c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "status": "Failed",
      "startTime": "2026-10-01T23:21:31.953266163Z",
      "endTime": "2026-10-01T23:40:37.297291005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: Creating: virtualmachine creating or updating\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv"
Unexpected error:
    <*fmt.wrapError | 0xc001076a40>: 
    failed to create NodePool np-bl9ftv: failed waiting for nodepool="np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv" in resourcegroup="rg-candidate-4-20-x4gs2n6gstps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "name": "c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "status": "Failed",
      "startTime": "2026-10-01T23:21:31.953266163Z",
      "endTime": "2026-10-01T23:40:37.297291005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: Creating: virtualmachine creating or updating\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009252c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-tfpfzw4vzd4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604",
      "name": "8e650b8c-c903-48db-ae29-20e1c0f74604",
      "status": "Failed",
      "startTime": "2026-10-01T23:01:41.16595227Z",
      "endTime": "2026-10-01T23:20:45.151714689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: Creating: virtualmachine creating or updating\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009252c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-tfpfzw4vzd4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604",
      "name": "8e650b8c-c903-48db-ae29-20e1c0f74604",
      "status": "Failed",
      "startTime": "2026-10-01T23:01:41.16595227Z",
      "endTime": "2026-10-01T23:20:45.151714689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: Creating: virtualmachine creating or updating\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-c4cdqr" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00120fcb0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-bpfkv76t7qfk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-c4cdqr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-c4cdqr" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00120fcb0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-bpfkv76t7qfk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-c4cdqr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-kwx4gf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014a3d10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-l8ml659lgnj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kwx4gf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-kwx4gf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014a3d10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-l8ml659lgnj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kwx4gf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-bngfn7" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000cf5770>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-q9xpw68pgwb7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-bngfn7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-bngfn7" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000cf5770>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-q9xpw68pgwb7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-bngfn7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-222nr2" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000993530>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-vf9snh5jjb8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-222nr2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-222nr2" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000993530>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-vf9snh5jjb8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-222nr2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr"
Unexpected error:
    <*fmt.wrapError | 0xc00081f300>: 
    failed to create NodePool np-8l66wr: failed waiting for nodepool="np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr" in resourcegroup="rg-candidate-4-20-66r4zk7wm6rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404",
      "name": "1513980f-1a90-4c5b-865f-c0a3b6601404",
      "status": "Failed",
      "startTime": "2026-10-01T03:20:14.931117191Z",
      "endTime": "2026-10-01T03:39:22.388104494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr"
Unexpected error:
    <*fmt.wrapError | 0xc00081f300>: 
    failed to create NodePool np-8l66wr: failed waiting for nodepool="np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr" in resourcegroup="rg-candidate-4-20-66r4zk7wm6rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404",
      "name": "1513980f-1a90-4c5b-865f-c0a3b6601404",
      "status": "Failed",
      "startTime": "2026-10-01T03:20:14.931117191Z",
      "endTime": "2026-10-01T03:39:22.388104494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-sq7dhp" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0015c4e70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-vjpbzl2jbr62/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-sq7dhp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-sq7dhp" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0015c4e70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-vjpbzl2jbr62/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-sq7dhp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc00171ed20>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-kc5k44mb955m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "name": "bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:20.214032414Z",
      "endTime": "2026-10-01T03:17:20.370385488Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-wwgqk-cz84n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-wwgqk-cz84n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc00171ed20>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-kc5k44mb955m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "name": "bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:20.214032414Z",
      "endTime": "2026-10-01T03:17:20.370385488Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-wwgqk-cz84n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-wwgqk-cz84n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-hwc6dp" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00164d620>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-txlxhwglc5dq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-hwc6dp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-hwc6dp" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00164d620>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-txlxhwglc5dq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-hwc6dp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-x5rqn9" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00163b1d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-d779fckrdm2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x5rqn9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-x5rqn9" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00163b1d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-d779fckrdm2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x5rqn9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-kbwzzs" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00104fe60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb6v87bszxhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-kbwzzs
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-kbwzzs" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00104fe60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb6v87bszxhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-kbwzzs
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-psr6l5" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001800d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-rn244qh65ldr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-psr6l5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-psr6l5" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001800d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-rn244qh65ldr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-psr6l5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-pr7dlf" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0013e6d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4ds2lzg2vct6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-pr7dlf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-pr7dlf" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0013e6d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4ds2lzg2vct6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-pr7dlf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d6ac40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-b7k2xmx45mmg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "name": "f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "status": "Failed",
      "startTime": "2026-10-01T02:57:30.013121006Z",
      "endTime": "2026-10-01T03:16:30.182282415Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d6ac40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-b7k2xmx45mmg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "name": "f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "status": "Failed",
      "startTime": "2026-10-01T02:57:30.013121006Z",
      "endTime": "2026-10-01T03:16:30.182282415Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000884ae0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-gcjj2967bz4p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "name": "8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "status": "Failed",
      "startTime": "2026-10-01T01:34:57.093691985Z",
      "endTime": "2026-10-01T01:54:01.709819246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000884ae0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-gcjj2967bz4p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "name": "8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "status": "Failed",
      "startTime": "2026-10-01T01:34:57.093691985Z",
      "endTime": "2026-10-01T01:54:01.709819246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred