alert fired 12 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:34:22Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kv8ld", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-kv8ld (namespace arobit) on cluster ci01-j7494016-svc has averaged 177.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:32:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b8f98bb8d-xxtkh", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b8f98bb8d-xxtkh (namespace mgmt-agent) on cluster ci01-j7494016-mgmt-1 has averaged 189.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:40:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8987ccb77-nmgwd", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8987ccb77-nmgwd (namespace clusters-service) on cluster ci01-j7494016-svc has averaged 256.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:29:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b8f98bb8d-l5m85", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b8f98bb8d-l5m85 (namespace mgmt-agent) on cluster ci01-j7494016-mgmt-2 has averaged 180% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:33:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-hmnhn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-hmnhn (namespace arobit) on cluster ci01-j7494016-mgmt-1 has averaged 217.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:29:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-d9thl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-d9thl (namespace arobit) on cluster ci01-j7494016-mgmt-2 has averaged 167.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:14:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-85694f5cf9-96x49", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-85694f5cf9-96x49 (namespace aro-hcp) on cluster ci01-j7494016-svc has averaged 418.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:40:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8987ccb77-btgnn", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8987ccb77-btgnn (namespace clusters-service) on cluster ci01-j7494016-svc has averaged 253.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:30:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-w55jk", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-w55jk (namespace arobit) on cluster ci01-j7494016-mgmt-1 has averaged 147.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:27:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-bwg62", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-bwg62 (namespace arobit) on cluster ci01-j7494016-svc has averaged 130.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:14:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8987ccb77-sfcv9", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8987ccb77-sfcv9 (namespace clusters-service) on cluster ci01-j7494016-svc has averaged 850.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:33:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-v5ccx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-v5ccx (namespace arobit) on cluster ci01-j7494016-mgmt-1 has averaged 160.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fcc7a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-jbw245668lwj" to finish creating: GET https://rp.j7559936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f",
      "name": "6d2a250e-0bac-4f44-a708-541cb00c701f",
      "status": "Failed",
      "startTime": "2026-09-16T23:34:27.803730917Z",
      "endTime": "2026-09-17T00:03:30.376925674Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fcc7a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-jbw245668lwj" to finish creating: GET https://rp.j7559936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f",
      "name": "6d2a250e-0bac-4f44-a708-541cb00c701f",
      "status": "Failed",
      "startTime": "2026-09-16T23:34:27.803730917Z",
      "endTime": "2026-09-17T00:03:30.376925674Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:46:37Z
  Ended:       2026-09-17T00:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7559936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdiuonvecl8bulup4napa18plli3hl-v5g6h1u7r3y6b9q", pod="router-77b5d8dffb-95c5m", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdiuonvecl8bulup4napa18plli3hl-v5g6h1u7r3y6b9q/router-77b5d8dffb-95c5m has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T23:46:37Z
  Ended:       2026-09-17T00:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7559936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdisk0mggfhm9130ika04epdikqlho-s5n1l0v6o6a7b8a", pod="router-55df5c6c57-b4j8r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdisk0mggfhm9130ika04epdikqlho-s5n1l0v6o6a7b8a/router-55df5c6c57-b4j8r has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T00:07:06Z
  Ended:       2026-09-17T00:23:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7559936-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-554b597df9-2phx7", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-554b597df9-2phx7 (namespace mgmt-agent) on cluster ci01-j7559936-mgmt-2 has averaged 182% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T00:07:06Z
  Ended:       2026-09-17T00:26:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7559936-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7ddmw", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-7ddmw (namespace arobit) on cluster ci01-j7559936-mgmt-2 has averaged 200.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:10:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2pvnx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2pvnx (namespace arobit) on cluster ci01-j3332864-mgmt-1 has averaged 186.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:10:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-dd7947c67-r26c4", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-dd7947c67-r26c4 (namespace mgmt-agent) on cluster ci01-j3332864-mgmt-1 has averaged 191.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:13:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-dd85c69cd-czx9t", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-dd85c69cd-czx9t (namespace clusters-service) on cluster ci01-j3332864-svc has averaged 238.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:13:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-558d75db97-9xbgb", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-558d75db97-9xbgb (namespace aro-hcp) on cluster ci01-j3332864-svc has averaged 416.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:13:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-72f4l", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-72f4l (namespace arobit) on cluster ci01-j3332864-svc has averaged 179.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-17T00:28:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-dd85c69cd-vfcn8", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-dd85c69cd-vfcn8 (namespace clusters-service) on cluster ci01-j3332864-svc has averaged 230.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-v9lkhvnzdhld/cluster-candidate-4-21-jrkc8h should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ba8a40>: 
    failed to create HCP cluster cluster-candidate-4-21-jrkc8h: failed waiting for cluster="cluster-candidate-4-21-jrkc8h" in resourcegroup="rg-candidate-4-21-v9lkhvnzdhld" to finish creating: GET https://rp.j8160640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38",
      "name": "61166157-eac2-4677-a0c1-58bf48e34b38",
      "status": "Failed",
      "startTime": "2026-09-16T23:26:49.985549857Z",
      "endTime": "2026-09-16T23:55:49.56140206Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-v9lkhvnzdhld/cluster-candidate-4-21-jrkc8h should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ba8a40>: 
    failed to create HCP cluster cluster-candidate-4-21-jrkc8h: failed waiting for cluster="cluster-candidate-4-21-jrkc8h" in resourcegroup="rg-candidate-4-21-v9lkhvnzdhld" to finish creating: GET https://rp.j8160640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38",
      "name": "61166157-eac2-4677-a0c1-58bf48e34b38",
      "status": "Failed",
      "startTime": "2026-09-16T23:26:49.985549857Z",
      "endTime": "2026-09-16T23:55:49.56140206Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T00:01:43Z
  Ended:       2026-09-17T00:07:42Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8160640-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8cxpn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8cxpn (namespace arobit) on cluster ci01-j8160640-mgmt-1 has averaged 131.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:38:16Z
  Ended:       2026-09-17T00:05:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8160640-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdff0k383bfd7grp5qnbnt5v4r1b0u-d3b2r2a4t0o4l6a", pod="router-5564cd696f-nb8mv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdff0k383bfd7grp5qnbnt5v4r1b0u-d3b2r2a4t0o4l6a/router-5564cd696f-nb8mv has been in a non-ready state for longer than 5 minutes.

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:51:00Z
  Ended:       2026-09-17T00:03:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3345536-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-9b88878b8-n2h2f", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-9b88878b8-n2h2f (namespace clusters-service) on cluster ci01-j3345536-svc has averaged 202% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T23:51:00Z
  Ended:       2026-09-17T00:03:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3345536-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rlqtg", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rlqtg (namespace arobit) on cluster ci01-j3345536-svc has averaged 160.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T23:51:00Z
  Ended:       2026-09-17T00:04:01Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3345536-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-9b88878b8-hnwhg", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-9b88878b8-hnwhg (namespace clusters-service) on cluster ci01-j3345536-svc has averaged 728% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T23:51:00Z
  Ended:       2026-09-17T00:03:01Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3345536-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-db5b7c885-ggk6l", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-db5b7c885-ggk6l (namespace aro-hcp) on cluster ci01-j3345536-svc has averaged 379.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T23:56:00Z
  Ended:       2026-09-17T00:05:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3345536-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-d9q79", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-d9q79 (namespace arobit) on cluster ci01-j3345536-mgmt-1 has averaged 130.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:31:05Z
  Ended:       2026-09-16T23:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3345536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdc08lq86mikr3m7qtfj8ulr4bs703-h7y5g4s0a4a5j6c", pod="router-5bb8b7c8bd-thvsf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdc08lq86mikr3m7qtfj8ulr4bs703-h7y5g4s0a4a5j6c/router-5bb8b7c8bd-thvsf has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T23:51:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8218240-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-579c9576fd-sv86s", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-579c9576fd-sv86s (namespace clusters-service) on cluster ci01-j8218240-svc has averaged 751.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T23:51:32Z
  Ended:       2026-09-16T23:57:37Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8218240-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-x6nxz", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-x6nxz (namespace arobit) on cluster ci01-j8218240-mgmt-2 has averaged 130.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T23:51:32Z
  Ended:       2026-09-16T23:57:37Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8218240-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-78464c98b7-mqbht", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-78464c98b7-mqbht (namespace mgmt-agent) on cluster ci01-j8218240-mgmt-2 has averaged 168.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f12a00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mt67hxmh7jr7" to finish creating: GET https://rp.j7063936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "name": "f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "status": "Failed",
      "startTime": "2026-09-16T22:42:15.998420325Z",
      "endTime": "2026-09-16T23:11:18.627710941Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f12a00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mt67hxmh7jr7" to finish creating: GET https://rp.j7063936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "name": "f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "status": "Failed",
      "startTime": "2026-09-16T22:42:15.998420325Z",
      "endTime": "2026-09-16T23:11:18.627710941Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f78b40>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bj966m2bsjb2" to finish creating: GET https://rp.j7063936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66342529-12fa-4da1-96e8-890f665b0b8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66342529-12fa-4da1-96e8-890f665b0b8e",
      "name": "66342529-12fa-4da1-96e8-890f665b0b8e",
      "status": "Failed",
      "startTime": "2026-09-16T22:39:31.439289121Z",
      "endTime": "2026-09-16T23:08:37.208910305Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 17m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f78b40>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bj966m2bsjb2" to finish creating: GET https://rp.j7063936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66342529-12fa-4da1-96e8-890f665b0b8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/66342529-12fa-4da1-96e8-890f665b0b8e",
      "name": "66342529-12fa-4da1-96e8-890f665b0b8e",
      "status": "Failed",
      "startTime": "2026-09-16T22:39:31.439289121Z",
      "endTime": "2026-09-16T23:08:37.208910305Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 17m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 14 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:11:46Z
  Ended:       2026-09-16T23:30:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-chksz", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-chksz (namespace arobit) on cluster ci01-j7063936-mgmt-1 has averaged 159.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T23:11:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5t5f6", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-5t5f6 (namespace arobit) on cluster ci01-j7063936-mgmt-2 has averaged 191.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T23:11:46Z
  Ended:       2026-09-16T23:36:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-554565c5c6-qnkg5", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-554565c5c6-qnkg5 (namespace mgmt-agent) on cluster ci01-j7063936-mgmt-2 has averaged 207.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T23:11:46Z
  Ended:       2026-09-16T23:38:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-557577f7c8-v7lhp", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-557577f7c8-v7lhp (namespace clusters-service) on cluster ci01-j7063936-svc has averaged 241.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T23:11:46Z
  Ended:       2026-09-16T23:35:44Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jqspw", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jqspw (namespace arobit) on cluster ci01-j7063936-svc has averaged 172.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T23:11:46Z
  Ended:       2026-09-16T23:37:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6wt6n", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6wt6n (namespace arobit) on cluster ci01-j7063936-mgmt-2 has averaged 203.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-16T23:11:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-557577f7c8-tk25g", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-557577f7c8-tk25g (namespace clusters-service) on cluster ci01-j7063936-svc has averaged 780.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-16T23:11:46Z
  Ended:       2026-09-16T23:30:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-554565c5c6-hz6w6", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-554565c5c6-hz6w6 (namespace mgmt-agent) on cluster ci01-j7063936-mgmt-1 has averaged 159.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-16T23:11:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-79554bdd98-8lr8l", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-79554bdd98-8lr8l (namespace aro-hcp) on cluster ci01-j7063936-svc has averaged 426.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-16T23:11:46Z
  Ended:       2026-09-16T23:38:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-557577f7c8-wdvtv", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-557577f7c8-wdvtv (namespace clusters-service) on cluster ci01-j7063936-svc has averaged 239.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-16T23:12:45Z
  Ended:       2026-09-16T23:29:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-wnbdn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-wnbdn (namespace arobit) on cluster ci01-j7063936-mgmt-2 has averaged 136.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-16T23:17:46Z
  Ended:       2026-09-16T23:23:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-wxbs2", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-wxbs2 (namespace arobit) on cluster ci01-j7063936-svc has averaged 121.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 13:
  State:       Resolved
  Started:     2026-09-16T23:20:46Z
  Ended:       2026-09-16T23:30:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-pfwsx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-pfwsx (namespace arobit) on cluster ci01-j7063936-mgmt-1 has averaged 123.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 14:
  State:       Resolved
  Started:     2026-09-16T23:21:44Z
  Ended:       2026-09-16T23:28:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7063936-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rtmxc", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rtmxc (namespace arobit) on cluster ci01-j7063936-mgmt-1 has averaged 121.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc0010967c0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-bfxzxpsnx2jt" to finish creating: GET https://rp.j4185600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12c613e0-b15b-45cf-943d-cff1110cd7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12c613e0-b15b-45cf-943d-cff1110cd7cf",
      "name": "12c613e0-b15b-45cf-943d-cff1110cd7cf",
      "status": "Failed",
      "startTime": "2026-09-16T22:29:03.123193133Z",
      "endTime": "2026-09-16T22:58:10.913482321Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc0010967c0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-bfxzxpsnx2jt" to finish creating: GET https://rp.j4185600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12c613e0-b15b-45cf-943d-cff1110cd7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12c613e0-b15b-45cf-943d-cff1110cd7cf",
      "name": "12c613e0-b15b-45cf-943d-cff1110cd7cf",
      "status": "Failed",
      "startTime": "2026-09-16T22:29:03.123193133Z",
      "endTime": "2026-09-16T22:58:10.913482321Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 8 time(s) (2 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T22:40:20Z
  Ended:       2026-09-16T22:57:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4185600-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssckc0qi9obj06817o08c2p0t18e9rl-v20261001", pod="router-7c865bd84c-4zb2h", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssckc0qi9obj06817o08c2p0t18e9rl-v20261001/router-7c865bd84c-4zb2h has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T23:03:18Z
  Ended:       2026-09-16T23:10:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4185600-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssckc0qi9obj06817o08c2p0t18e9rl-v20261001", pod="router-7c865bd84c-4zb2h", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssckc0qi9obj06817o08c2p0t18e9rl-v20261001/router-7c865bd84c-4zb2h has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:03:25Z
  Ended:       2026-09-16T23:10:27Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4185600-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-57644fb98d-2b9lt", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-57644fb98d-2b9lt (namespace aro-hcp) on cluster ci01-j4185600-svc has averaged 418.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T23:03:25Z
  Ended:       2026-09-16T23:10:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4185600-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-6d579cc9b7-wks59", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-6d579cc9b7-wks59 (namespace clusters-service) on cluster ci01-j4185600-svc has averaged 243.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T23:03:25Z
  Ended:       2026-09-16T23:10:28Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4185600-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-cs2rw", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-cs2rw (namespace arobit) on cluster ci01-j4185600-svc has averaged 177.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:08:05Z
  Ended:       2026-09-16T23:20:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-68bbb68cfb-4n82q", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-68bbb68cfb-4n82q (namespace clusters-service) on cluster ci01-j9343616-svc has averaged 685.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T23:09:03Z
  Ended:       2026-09-16T23:21:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-68bbb68cfb-dlbqw", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-68bbb68cfb-dlbqw (namespace clusters-service) on cluster ci01-j9343616-svc has averaged 214.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T23:11:06Z
  Ended:       2026-09-16T23:24:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-b4rjl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-b4rjl (namespace arobit) on cluster ci01-j9343616-mgmt-1 has averaged 217.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T23:11:06Z
  Ended:       2026-09-16T23:22:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6bc4f4cb68-nlpz6", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6bc4f4cb68-nlpz6 (namespace mgmt-agent) on cluster ci01-j9343616-mgmt-2 has averaged 141.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T23:13:03Z
  Ended:       2026-09-16T23:29:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-68bbb68cfb-5jm25", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-68bbb68cfb-5jm25 (namespace clusters-service) on cluster ci01-j9343616-svc has averaged 228.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T23:13:03Z
  Ended:       2026-09-16T23:29:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fnbdj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fnbdj (namespace arobit) on cluster ci01-j9343616-svc has averaged 163.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-16T23:13:03Z
  Ended:       2026-09-16T23:29:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-59d56584b6-7vtjg", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-59d56584b6-7vtjg (namespace aro-hcp) on cluster ci01-j9343616-svc has averaged 395.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-16T23:13:03Z
  Ended:       2026-09-16T23:35:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9343616-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6bc4f4cb68-dmrjw", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6bc4f4cb68-dmrjw (namespace mgmt-agent) on cluster ci01-j9343616-mgmt-1 has averaged 182.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T22:36:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6960256-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-64cc5944b6-97trg", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-64cc5944b6-97trg (namespace clusters-service) on cluster ci01-j6960256-svc has averaged 807.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T22:36:48Z
  Ended:       2026-09-16T22:50:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6960256-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5v5zv", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-5v5zv (namespace arobit) on cluster ci01-j6960256-mgmt-1 has averaged 130.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T22:36:48Z
  Ended:       2026-09-16T22:47:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6960256-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5cbb7b949c-wd62b", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5cbb7b949c-wd62b (namespace mgmt-agent) on cluster ci01-j6960256-mgmt-2 has averaged 139.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T22:36:48Z
  Ended:       2026-09-16T22:49:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6960256-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-w9rm6", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-w9rm6 (namespace arobit) on cluster ci01-j6960256-mgmt-2 has averaged 156.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T22:12:55Z
  Ended:       2026-09-16T22:18:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6960256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssc7miveafqg7ekrqti3nqvne3cgsn6-m6t6u0i7x2h4r2q", pod="router-5f6867d758-4rblw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssc7miveafqg7ekrqti3nqvne3cgsn6-m6t6u0i7x2h4r2q/router-5f6867d758-4rblw has been in a non-ready state for longer than 5 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T22:57:46Z
  Ended:       2026-09-16T23:15:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0952320-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5984h", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-5984h (namespace arobit) on cluster ci01-j0952320-mgmt-2 has averaged 241.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T22:57:46Z
  Ended:       2026-09-16T23:04:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0952320-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gr7jc", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gr7jc (namespace arobit) on cluster ci01-j0952320-mgmt-2 has averaged 146.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T22:57:46Z
  Ended:       2026-09-16T23:05:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0952320-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-qff9w", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-qff9w (namespace arobit) on cluster ci01-j0952320-mgmt-2 has averaged 124.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T22:57:46Z
  Ended:       2026-09-16T23:14:48Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0952320-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-78ff4b7d87-grnwc", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-78ff4b7d87-grnwc (namespace mgmt-agent) on cluster ci01-j0952320-mgmt-2 has averaged 247.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

time=2026-09-16T21:39:23.882Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T21:39:24.651Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-16T21:39:25.419Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=1957e733d563fe8d01011d2966a166813e9314307467a323647f090022c9d7a7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1475712-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F1957e733d563fe8d01011d2966a166813e9314307467a323647f090022c9d7a7
time=2026-09-16T22:09:23.556Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-dgkdj", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-dgkdj has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-cb282", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-cb282 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-hdnbp", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-hdnbp has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-f58cs", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-f58cs has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T22:06:04Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.104:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1475712/providers/microsoft.monitor/accounts/services-j1475712", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-59mfc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T22:06:04Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.104:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1475712/providers/microsoft.monitor/accounts/services-j1475712", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-59mfc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T22:05:46Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j1475712-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j1475712-svc has been unreachable for 15 minutes.

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T22:15:49Z
  Ended:       2026-09-16T22:30:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4784256-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5ff76f7dfd-8gkdt", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5ff76f7dfd-8gkdt (namespace mgmt-agent) on cluster ci01-j4784256-mgmt-1 has averaged 179.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T22:15:49Z
  Ended:       2026-09-16T22:31:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4784256-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6m54g", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6m54g (namespace arobit) on cluster ci01-j4784256-mgmt-2 has averaged 143.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T22:15:49Z
  Ended:       2026-09-16T22:37:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4784256-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7bfb85644-ctddh", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7bfb85644-ctddh (namespace clusters-service) on cluster ci01-j4784256-svc has averaged 238.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T22:15:49Z
  Ended:       2026-09-16T22:27:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4784256-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jts9j", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jts9j (namespace arobit) on cluster ci01-j4784256-mgmt-2 has averaged 255.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T22:15:49Z
  Ended:       2026-09-16T22:41:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4784256-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7bfb85644-6sh92", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7bfb85644-6sh92 (namespace clusters-service) on cluster ci01-j4784256-svc has averaged 287.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T22:02:06Z
  Ended:       2026-09-16T22:09:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5185280-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7bb74f5747-b8mnf", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7bb74f5747-b8mnf (namespace mgmt-agent) on cluster ci01-j5185280-mgmt-2 has averaged 158.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T22:02:06Z
  Ended:       2026-09-16T22:08:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5185280-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-68b58f9664-thkdd", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-68b58f9664-thkdd (namespace clusters-service) on cluster ci01-j5185280-svc has averaged 247.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T22:02:07Z
  Ended:       2026-09-16T22:09:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5185280-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-n5mgg", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-n5mgg (namespace arobit) on cluster ci01-j5185280-mgmt-2 has averaged 146.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T22:06:02Z
  Ended:       2026-09-16T22:32:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0645888-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-55fbb6c9-ccknz", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-55fbb6c9-ccknz (namespace aro-hcp) on cluster ci01-j0645888-svc has averaged 419.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T22:06:02Z
  Ended:       2026-09-16T22:32:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0645888-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-586c94d488-qmqpw", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-586c94d488-qmqpw (namespace clusters-service) on cluster ci01-j0645888-svc has averaged 776.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T22:06:02Z
  Ended:       2026-09-16T22:20:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0645888-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-586c94d488-kqxfq", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-586c94d488-kqxfq (namespace clusters-service) on cluster ci01-j0645888-svc has averaged 232.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T22:06:02Z
  Ended:       2026-09-16T22:29:59Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0645888-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-42shp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-42shp (namespace arobit) on cluster ci01-j0645888-svc has averaged 224% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:54:30Z
  Ended:       2026-09-16T22:00:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0645888-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbr3b0dohpb942farb5mg8qfbn5fpn-v9m4k2a5i1j6u4q", pod="openshift-apiserver-698dfb475d-rm5kz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbr3b0dohpb942farb5mg8qfbn5fpn-v9m4k2a5i1j6u4q/openshift-apiserver-698dfb475d-rm5kz has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-n9lbrg2g9s97/cluster-candidate-5-0-qbdt9b should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fac960>: 
    failed to create HCP cluster cluster-candidate-5-0-qbdt9b: failed waiting for cluster="cluster-candidate-5-0-qbdt9b" in resourcegroup="rg-candidate-5-0-n9lbrg2g9s97" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d8dfefa-03ec-411c-ae11-a19a19b9c715
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d8dfefa-03ec-411c-ae11-a19a19b9c715",
      "name": "1d8dfefa-03ec-411c-ae11-a19a19b9c715",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:26.135723838Z",
      "endTime": "2026-09-16T21:51:33.47561355Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 21m49s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-n9lbrg2g9s97/cluster-candidate-5-0-qbdt9b should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fac960>: 
    failed to create HCP cluster cluster-candidate-5-0-qbdt9b: failed waiting for cluster="cluster-candidate-5-0-qbdt9b" in resourcegroup="rg-candidate-5-0-n9lbrg2g9s97" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d8dfefa-03ec-411c-ae11-a19a19b9c715
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d8dfefa-03ec-411c-ae11-a19a19b9c715",
      "name": "1d8dfefa-03ec-411c-ae11-a19a19b9c715",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:26.135723838Z",
      "endTime": "2026-09-16T21:51:33.47561355Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 21m49s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0013ea750>: 
    GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd13f769-88f3-4db4-9a16-30099a1f1771
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd13f769-88f3-4db4-9a16-30099a1f1771",
      "name": "dd13f769-88f3-4db4-9a16-30099a1f1771",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:59.852105636Z",
      "endTime": "2026-09-16T21:52:03.393836187Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0013ea750>: 
    GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd13f769-88f3-4db4-9a16-30099a1f1771
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dd13f769-88f3-4db4-9a16-30099a1f1771",
      "name": "dd13f769-88f3-4db4-9a16-30099a1f1771",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:59.852105636Z",
      "endTime": "2026-09-16T21:52:03.393836187Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000614ae0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-nbncrnhpbr7g" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e5d0092-fd65-4a3b-8b4d-9aa58e610368
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e5d0092-fd65-4a3b-8b4d-9aa58e610368",
      "name": "2e5d0092-fd65-4a3b-8b4d-9aa58e610368",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:37.508203008Z",
      "endTime": "2026-09-16T21:51:43.476984266Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000614ae0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-nbncrnhpbr7g" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e5d0092-fd65-4a3b-8b4d-9aa58e610368
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2e5d0092-fd65-4a3b-8b4d-9aa58e610368",
      "name": "2e5d0092-fd65-4a3b-8b4d-9aa58e610368",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:37.508203008Z",
      "endTime": "2026-09-16T21:51:43.476984266Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:183]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:183]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:89]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000c1e780>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-pzvkp84k7zw9" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cfbae6c-9a9c-457e-badc-6962e84d4e6c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cfbae6c-9a9c-457e-badc-6962e84d4e6c",
      "name": "9cfbae6c-9a9c-457e-badc-6962e84d4e6c",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:37.245133028Z",
      "endTime": "2026-09-16T21:51:43.591474051Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:89]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000c1e780>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-pzvkp84k7zw9" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cfbae6c-9a9c-457e-badc-6962e84d4e6c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9cfbae6c-9a9c-457e-badc-6962e84d4e6c",
      "name": "9cfbae6c-9a9c-457e-badc-6962e84d4e6c",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:37.245133028Z",
      "endTime": "2026-09-16T21:51:43.591474051Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:99]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000aa2500>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-87r4ztg6n9l7" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d00f648b-166f-4d0d-b942-191a0be88942
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d00f648b-166f-4d0d-b942-191a0be88942",
      "name": "d00f648b-166f-4d0d-b942-191a0be88942",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:41.289177596Z",
      "endTime": "2026-09-16T21:51:43.506899271Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:99]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000aa2500>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-87r4ztg6n9l7" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d00f648b-166f-4d0d-b942-191a0be88942
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d00f648b-166f-4d0d-b942-191a0be88942",
      "name": "d00f648b-166f-4d0d-b942-191a0be88942",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:41.289177596Z",
      "endTime": "2026-09-16T21:51:43.506899271Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: failed to create HCP cluster "cluster-zstream-5-0-9bj8v9" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000aaa440>: 
    failed to create HCP cluster cluster-zstream-5-0-9bj8v9: failed waiting for cluster="cluster-zstream-5-0-9bj8v9" in resourcegroup="rg-zstream-upgrade-5-0-km48vzh4hvjm" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd9c2ad0-6041-47d2-bcf5-57e68c17fd53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd9c2ad0-6041-47d2-bcf5-57e68c17fd53",
      "name": "bd9c2ad0-6041-47d2-bcf5-57e68c17fd53",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:40.065777395Z",
      "endTime": "2026-09-16T21:51:43.490166655Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.1 is Partial (want Completed), started 22m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: failed to create HCP cluster "cluster-zstream-5-0-9bj8v9" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000aaa440>: 
    failed to create HCP cluster cluster-zstream-5-0-9bj8v9: failed waiting for cluster="cluster-zstream-5-0-9bj8v9" in resourcegroup="rg-zstream-upgrade-5-0-km48vzh4hvjm" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd9c2ad0-6041-47d2-bcf5-57e68c17fd53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd9c2ad0-6041-47d2-bcf5-57e68c17fd53",
      "name": "bd9c2ad0-6041-47d2-bcf5-57e68c17fd53",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:40.065777395Z",
      "endTime": "2026-09-16T21:51:43.490166655Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.1 is Partial (want Completed), started 22m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:90]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000eaac80>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-5sd5p2zrd22p" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb",
      "name": "cb287309-affe-4cb6-ba60-0ab97101fffb",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:15.559693502Z",
      "endTime": "2026-09-16T21:51:23.369287505Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:90]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000eaac80>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-5sd5p2zrd22p" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb",
      "name": "cb287309-affe-4cb6-ba60-0ab97101fffb",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:15.559693502Z",
      "endTime": "2026-09-16T21:51:23.369287505Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:293]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000adcb40>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8b5zl2hzsmbj" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/80bc4253-f3f0-43bc-a084-2e6ad2732f13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/80bc4253-f3f0-43bc-a084-2e6ad2732f13",
      "name": "80bc4253-f3f0-43bc-a084-2e6ad2732f13",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:42.844869164Z",
      "endTime": "2026-09-16T21:51:43.481872302Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:293]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000adcb40>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8b5zl2hzsmbj" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/80bc4253-f3f0-43bc-a084-2e6ad2732f13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/80bc4253-f3f0-43bc-a084-2e6ad2732f13",
      "name": "80bc4253-f3f0-43bc-a084-2e6ad2732f13",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:42.844869164Z",
      "endTime": "2026-09-16T21:51:43.481872302Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:85]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001056d00>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-c7nrjx2xztfs" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a484a274-602c-4541-905e-dc02a0a3186b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a484a274-602c-4541-905e-dc02a0a3186b",
      "name": "a484a274-602c-4541-905e-dc02a0a3186b",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:10.633199439Z",
      "endTime": "2026-09-16T21:51:13.479553961Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:85]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001056d00>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-c7nrjx2xztfs" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a484a274-602c-4541-905e-dc02a0a3186b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a484a274-602c-4541-905e-dc02a0a3186b",
      "name": "a484a274-602c-4541-905e-dc02a0a3186b",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:10.633199439Z",
      "endTime": "2026-09-16T21:51:13.479553961Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 11 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:35:38Z
  Ended:       2026-09-16T22:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbld7o605mco1c6q9ktqjhs93eqe6s-j8l1v5a2x0c6s4f", pod="router-6bcf688cbb-nxnjs", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbld7o605mco1c6q9ktqjhs93eqe6s-j8l1v5a2x0c6s4f/router-6bcf688cbb-nxnjs has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:35:38Z
  Ended:       2026-09-16T21:59:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl8nc6r8al7si3mq1rjgliso80a2c-ephemeral-disk", pod="router-6b8476f84b-vhtcn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl8nc6r8al7si3mq1rjgliso80a2c-ephemeral-disk/router-6b8476f84b-vhtcn has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T22:03:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl0m0uhvtufc2gt2visu064e21mub-u5h1i4b8r4n0e8j", pod="router-7b47b96958-mgtjv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl0m0uhvtufc2gt2visu064e21mub-u5h1i4b8r4n0e8j/router-7b47b96958-mgtjv has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T21:59:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl4s13qkvde5phnn7127rdghp4v37-t0q4t2z8c3u3y8r", pod="router-58bf477cdf-kpghv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl4s13qkvde5phnn7127rdghp4v37-t0q4t2z8c3u3y8r/router-58bf477cdf-kpghv has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T22:07:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl7klscstpk49ij7po527rbns6pgi-p3b0r5x5e4p9u1v", pod="router-5745db7ddf-xsgg8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl7klscstpk49ij7po527rbns6pgi-p3b0r5x5e4p9u1v/router-5745db7ddf-xsgg8 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T22:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl966ue7bj7p60s8sinf6qmbqnuok-h7n4x2g5p8k0m4i", pod="router-5578b5cd64-55cjc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl966ue7bj7p60s8sinf6qmbqnuok-h7n4x2g5p8k0m4i/router-5578b5cd64-55cjc has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T21:59:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl28cqnaa6fvtqs8lmbcrn1lmg4a5-b7b7m9f6p3w4f2s", pod="router-69d54c7db5-fvcnx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl28cqnaa6fvtqs8lmbcrn1lmg4a5-b7b7m9f6p3w4f2s/router-69d54c7db5-fvcnx has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T22:07:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl7bvf4r5c6jetumc9dq8tqgsfr18-cilium-cl", pod="router-7b8cc969f7-4pr9f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl7bvf4r5c6jetumc9dq8tqgsfr18-cilium-cl/router-7b8cc969f7-4pr9f has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T21:59:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl81hl56coj0npht1eplmr0rhedd6-d1c8e6y4l1z3b4z", pod="router-654684649f-ztg2t", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl81hl56coj0npht1eplmr0rhedd6-d1c8e6y4l1z3b4z/router-654684649f-ztg2t has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T21:58:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl5466uev4mojug68na19t1k1bhpg-j2c1q0a4t2i8o4k", pod="router-75c68fb5b9-cfmtm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl5466uev4mojug68na19t1k1bhpg-j2c1q0a4t2i8o4k/router-75c68fb5b9-cfmtm has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-16T21:36:40Z
  Ended:       2026-09-16T22:05:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3686400-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbl0c4vr90is2erighv55thdli5lnq-m4n7d3j0c0g6t2h", pod="router-86c7ffd6cc-mhnb6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbl0c4vr90is2erighv55thdli5lnq-m4n7d3j0c0g6t2h/router-86c7ffd6cc-mhnb6 has been in a non-ready state for longer than 5 minutes.

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:56:22Z
  Ended:       2026-09-16T22:07:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3686400-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-cnwz7", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-cnwz7 (namespace arobit) on cluster ci01-j3686400-mgmt-2 has averaged 231.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:56:22Z
  Ended:       2026-09-16T22:07:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3686400-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-66bbcd5d66-jbqlh", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-66bbcd5d66-jbqlh (namespace clusters-service) on cluster ci01-j3686400-svc has averaged 997.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T21:56:22Z
  Ended:       2026-09-16T22:07:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3686400-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-67b9d68855-97z44", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-67b9d68855-97z44 (namespace mgmt-agent) on cluster ci01-j3686400-mgmt-2 has averaged 165.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T21:56:22Z
  Ended:       2026-09-16T22:22:23Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3686400-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2sxwz", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2sxwz (namespace arobit) on cluster ci01-j3686400-svc has averaged 253.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T21:57:22Z
  Ended:       2026-09-16T22:07:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3686400-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-sxn79", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-sxn79 (namespace arobit) on cluster ci01-j3686400-svc has averaged 137% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T21:58:21Z
  Ended:       2026-09-16T22:22:22Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3686400-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-66bbcd5d66-l7s2g", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-66bbcd5d66-l7s2g (namespace clusters-service) on cluster ci01-j3686400-svc has averaged 249.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:58:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3686400-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-665d88bb7f-br5fz", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-665d88bb7f-br5fz (namespace aro-hcp) on cluster ci01-j3686400-svc has averaged 424.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-dk8z27cq8m2q/cluster-candidate-5-0-lt55lh should provision
Unexpected error:
    <*fmt.wrapError | 0xc00088abe0>: 
    failed to create HCP cluster cluster-candidate-5-0-lt55lh: failed waiting for cluster="cluster-candidate-5-0-lt55lh" in resourcegroup="rg-candidate-5-0-dk8z27cq8m2q" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5a4fb94-5cbc-41ce-a664-6b95d7517b0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5a4fb94-5cbc-41ce-a664-6b95d7517b0f",
      "name": "c5a4fb94-5cbc-41ce-a664-6b95d7517b0f",
      "status": "Failed",
      "startTime": "2026-09-16T21:07:59.319124626Z",
      "endTime": "2026-09-16T21:37:03.975119557Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 25m26s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-dk8z27cq8m2q/cluster-candidate-5-0-lt55lh should provision
Unexpected error:
    <*fmt.wrapError | 0xc00088abe0>: 
    failed to create HCP cluster cluster-candidate-5-0-lt55lh: failed waiting for cluster="cluster-candidate-5-0-lt55lh" in resourcegroup="rg-candidate-5-0-dk8z27cq8m2q" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5a4fb94-5cbc-41ce-a664-6b95d7517b0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5a4fb94-5cbc-41ce-a664-6b95d7517b0f",
      "name": "c5a4fb94-5cbc-41ce-a664-6b95d7517b0f",
      "status": "Failed",
      "startTime": "2026-09-16T21:07:59.319124626Z",
      "endTime": "2026-09-16T21:37:03.975119557Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 25m26s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc002923158>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node o2d9i5v3g7d8o6s-np-1-9rk5g-gzzc5: failed to get pod logs for pod fips-check-psxxs in namespace e2e-fips-test-rx7hl on node o2d9i5v3g7d8o6s-np-1-9rk5g-gzzc5 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-psxxs)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc002923158>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node o2d9i5v3g7d8o6s-np-1-9rk5g-gzzc5: failed to get pod logs for pod fips-check-psxxs in namespace e2e-fips-test-rx7hl on node o2d9i5v3g7d8o6s-np-1-9rk5g-gzzc5 after retries: an error on the server ("unknown") has prevented the request from succeeding (get pods fips-check-psxxs)
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f849a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-kxnvbkzf9rsh" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56e26800-2105-4cf7-a6ab-3433abf35800
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56e26800-2105-4cf7-a6ab-3433abf35800",
      "name": "56e26800-2105-4cf7-a6ab-3433abf35800",
      "status": "Failed",
      "startTime": "2026-09-16T21:09:44.457930894Z",
      "endTime": "2026-09-16T21:38:44.111074819Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f849a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-kxnvbkzf9rsh" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56e26800-2105-4cf7-a6ab-3433abf35800
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56e26800-2105-4cf7-a6ab-3433abf35800",
      "name": "56e26800-2105-4cf7-a6ab-3433abf35800",
      "status": "Failed",
      "startTime": "2026-09-16T21:09:44.457930894Z",
      "endTime": "2026-09-16T21:38:44.111074819Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m11s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011165a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2nsvjcl7j6m6" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0",
      "name": "a27b380d-13cc-4abe-9932-546cf6483df0",
      "status": "Failed",
      "startTime": "2026-09-16T21:08:44.335887928Z",
      "endTime": "2026-09-16T21:37:44.000335858Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011165a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2nsvjcl7j6m6" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0",
      "name": "a27b380d-13cc-4abe-9932-546cf6483df0",
      "status": "Failed",
      "startTime": "2026-09-16T21:08:44.335887928Z",
      "endTime": "2026-09-16T21:37:44.000335858Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e75200>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dfv98dd4pvxk" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ad1b2fe-b261-4381-9290-0e87a2df230a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ad1b2fe-b261-4381-9290-0e87a2df230a",
      "name": "5ad1b2fe-b261-4381-9290-0e87a2df230a",
      "status": "Failed",
      "startTime": "2026-09-16T21:07:46.906089867Z",
      "endTime": "2026-09-16T21:36:54.029333318Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e75200>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dfv98dd4pvxk" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ad1b2fe-b261-4381-9290-0e87a2df230a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ad1b2fe-b261-4381-9290-0e87a2df230a",
      "name": "5ad1b2fe-b261-4381-9290-0e87a2df230a",
      "status": "Failed",
      "startTime": "2026-09-16T21:07:46.906089867Z",
      "endTime": "2026-09-16T21:36:54.029333318Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vtt65b86m6vb/cluster-candidate-4-21-gj8xr2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c20920>: 
    failed to create HCP cluster cluster-candidate-4-21-gj8xr2: failed waiting for cluster="cluster-candidate-4-21-gj8xr2" in resourcegroup="rg-candidate-4-21-vtt65b86m6vb" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bcaf370-2958-4c5f-97cb-f96058b55495
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bcaf370-2958-4c5f-97cb-f96058b55495",
      "name": "9bcaf370-2958-4c5f-97cb-f96058b55495",
      "status": "Failed",
      "startTime": "2026-09-16T21:08:12.529393727Z",
      "endTime": "2026-09-16T21:37:13.94502458Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vtt65b86m6vb/cluster-candidate-4-21-gj8xr2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c20920>: 
    failed to create HCP cluster cluster-candidate-4-21-gj8xr2: failed waiting for cluster="cluster-candidate-4-21-gj8xr2" in resourcegroup="rg-candidate-4-21-vtt65b86m6vb" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bcaf370-2958-4c5f-97cb-f96058b55495
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9bcaf370-2958-4c5f-97cb-f96058b55495",
      "name": "9bcaf370-2958-4c5f-97cb-f96058b55495",
      "status": "Failed",
      "startTime": "2026-09-16T21:08:12.529393727Z",
      "endTime": "2026-09-16T21:37:13.94502458Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:40:02Z
  Ended:       2026-09-16T21:52:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3803776-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2lktz", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2lktz (namespace arobit) on cluster ci01-j3803776-mgmt-1 has averaged 137.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:40:02Z
  Ended:       2026-09-16T21:54:03Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3803776-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-99f86f48f-d6hvl", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-99f86f48f-d6hvl (namespace mgmt-agent) on cluster ci01-j3803776-mgmt-1 has averaged 164.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:41:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3803776-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6644cd5d7d-cfxst", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6644cd5d7d-cfxst (namespace aro-hcp) on cluster ci01-j3803776-svc has averaged 423.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:41:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3803776-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5454d5ff57-pdkgp", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5454d5ff57-pdkgp (namespace clusters-service) on cluster ci01-j3803776-svc has averaged 829.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:41:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3803776-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-b5mqv", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-b5mqv (namespace arobit) on cluster ci01-j3803776-svc has averaged 235.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:44:54Z
  Ended:       2026-09-16T21:51:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3803776-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.109:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3803776/providers/microsoft.monitor/accounts/hcps-j3803776", namespace="ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r", pod="arohcp-monitor-kube-state-metrics-574f657d96-gndsj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:45:52Z
  Ended:       2026-09-16T22:01:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3803776-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.109:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3803776/providers/microsoft.monitor/accounts/hcps-j3803776", namespace="ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r", pod="arohcp-monitor-kube-state-metrics-574f657d96-gndsj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s) (3 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:17:53Z
  Ended:       2026-09-16T21:52:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3803776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbec6gci69870qqvi16u5kcapftr0i-j9u1q2u4g1r8a9m", pod="router-74f7d556c5-zbmxx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbec6gci69870qqvi16u5kcapftr0i-j9u1q2u4g1r8a9m/router-74f7d556c5-zbmxx has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:18:51Z
  Ended:       2026-09-16T21:50:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3803776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbe8pmlnu9ao5hdglrhgldjpnqc79l-k0m6y3i5a2o3n4m", pod="router-987d55cf6-zxs7p", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbe8pmlnu9ao5hdglrhgldjpnqc79l-k0m6y3i5a2o3n4m/router-987d55cf6-zxs7p has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T21:19:51Z
  Ended:       2026-09-16T21:52:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3803776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssbeffd09kboib4diqvu4luf8fh3hjb-m7i6y4e0g4j7v8c", pod="router-5fccd44c6d-zgbxq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssbeffd09kboib4diqvu4luf8fh3hjb-m7i6y4e0g4j7v8c/router-5fccd44c6d-zgbxq has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000edc320>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-d77hgbbbw6rf" to finish creating: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd",
      "name": "74e66356-25b3-4d6f-95af-f1292008b6fd",
      "status": "Failed",
      "startTime": "2026-09-16T20:47:22.69341748Z",
      "endTime": "2026-09-16T21:16:24.116227211Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 6m20s ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000edc320>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-d77hgbbbw6rf" to finish creating: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd",
      "name": "74e66356-25b3-4d6f-95af-f1292008b6fd",
      "status": "Failed",
      "startTime": "2026-09-16T20:47:22.69341748Z",
      "endTime": "2026-09-16T21:16:24.116227211Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 6m20s ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001400078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-dqgbxbhrnvkj" to finish deleting: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b",
      "name": "38120612-3f62-4d83-b1c9-f450e308488b",
      "status": "Failed",
      "startTime": "2026-09-16T20:57:03.974907545Z",
      "endTime": "2026-09-16T21:21:04.12131988Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssb46njkre58n4p9mol1i33anuaqjtv still exists (deletion dispatched at 2026-09-16T20:57:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001400078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-dqgbxbhrnvkj" to finish deleting: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b",
      "name": "38120612-3f62-4d83-b1c9-f450e308488b",
      "status": "Failed",
      "startTime": "2026-09-16T20:57:03.974907545Z",
      "endTime": "2026-09-16T21:21:04.12131988Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssb46njkre58n4p9mol1i33anuaqjtv still exists (deletion dispatched at 2026-09-16T20:57:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:20:58Z
  Ended:       2026-09-16T21:49:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6676096-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7d575bbb5-wsp69", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7d575bbb5-wsp69 (namespace clusters-service) on cluster ci01-j6676096-svc has averaged 260.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:20:58Z
  Ended:       2026-09-16T21:38:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6676096-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-pvsvr", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-pvsvr (namespace arobit) on cluster ci01-j6676096-mgmt-1 has averaged 141.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:28:57Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6676096-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7czlv", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-7czlv (namespace arobit) on cluster ci01-j6676096-mgmt-1 has averaged 186.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008e6240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-x4ht6zckdvtk" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab",
      "name": "675795c0-1d48-4443-bf17-96b820fb9dab",
      "status": "Failed",
      "startTime": "2026-09-16T20:56:31.8857491Z",
      "endTime": "2026-09-16T21:20:37.108128013Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssari2k27ic1890mj0nf8obfposs9em still exists (deletion dispatched at 2026-09-16T20:56:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008e6240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-x4ht6zckdvtk" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab",
      "name": "675795c0-1d48-4443-bf17-96b820fb9dab",
      "status": "Failed",
      "startTime": "2026-09-16T20:56:31.8857491Z",
      "endTime": "2026-09-16T21:20:37.108128013Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssari2k27ic1890mj0nf8obfposs9em still exists (deletion dispatched at 2026-09-16T20:56:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019540a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-kttjnq" in resourcegroup="rg-np-downgrade-kttjnq-75tphq26mtkt" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/941782dd-efae-4093-940b-ca21585fc901
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/941782dd-efae-4093-940b-ca21585fc901",
      "name": "941782dd-efae-4093-940b-ca21585fc901",
      "status": "Failed",
      "startTime": "2026-09-16T20:46:21.836683685Z",
      "endTime": "2026-09-16T21:10:27.160876667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssarr47iva7e9en1e12mbat1fvbbvpq still exists (deletion dispatched at 2026-09-16T20:46:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019540a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-kttjnq" in resourcegroup="rg-np-downgrade-kttjnq-75tphq26mtkt" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/941782dd-efae-4093-940b-ca21585fc901
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/941782dd-efae-4093-940b-ca21585fc901",
      "name": "941782dd-efae-4093-940b-ca21585fc901",
      "status": "Failed",
      "startTime": "2026-09-16T20:46:21.836683685Z",
      "endTime": "2026-09-16T21:10:27.160876667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssarr47iva7e9en1e12mbat1fvbbvpq still exists (deletion dispatched at 2026-09-16T20:46:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001910318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-tfkgg85wrhjp" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51",
      "name": "40b6c313-a567-4259-bb1a-298290750f51",
      "status": "Failed",
      "startTime": "2026-09-16T20:55:14.718059601Z",
      "endTime": "2026-09-16T21:19:17.098694741Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssarn8fr2snie5k70mjfefs0cg1miti still exists (deletion dispatched at 2026-09-16T20:55:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001910318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-tfkgg85wrhjp" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51",
      "name": "40b6c313-a567-4259-bb1a-298290750f51",
      "status": "Failed",
      "startTime": "2026-09-16T20:55:14.718059601Z",
      "endTime": "2026-09-16T21:19:17.098694741Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssarn8fr2snie5k70mjfefs0cg1miti still exists (deletion dispatched at 2026-09-16T20:55:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-update-nodes: failed waiting for nodepool="np-update-nodes" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ckch4xg6nxh4" to finish creating: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84",
  "name": "e55eeddb-0ead-432e-a110-c696080abf84",
  "status": "Failed",
  "startTime": "2026-09-16T20:39:13.615216045Z",
  "endTime": "2026-09-16T20:58:16.977517435Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-update-nodes: failed waiting for nodepool="np-update-nodes" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ckch4xg6nxh4" to finish creating: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84",
  "name": "e55eeddb-0ead-432e-a110-c696080abf84",
  "status": "Failed",
  "startTime": "2026-09-16T20:39:13.615216045Z",
  "endTime": "2026-09-16T20:58:16.977517435Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:02:59Z
  Ended:       2026-09-16T21:14:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6757888-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jmb7z", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jmb7z (namespace arobit) on cluster ci01-j6757888-mgmt-1 has averaged 165.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:02:59Z
  Ended:       2026-09-16T21:18:01Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6757888-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-f62zl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-f62zl (namespace arobit) on cluster ci01-j6757888-mgmt-2 has averaged 179.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T21:02:59Z
  Ended:       2026-09-16T21:09:00Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6757888-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-844597f54-s6sj8", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-844597f54-s6sj8 (namespace clusters-service) on cluster ci01-j6757888-svc has averaged 249.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T21:02:59Z
  Ended:       2026-09-16T21:19:01Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6757888-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-89b689549-csthj", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-89b689549-csthj (namespace mgmt-agent) on cluster ci01-j6757888-mgmt-2 has averaged 199.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:38:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j6757888-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-bcnth", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-bcnth (namespace arobit) on cluster ci01-j6757888-svc has averaged 127.2% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T20:41:10Z
  Ended:       2026-09-16T21:03:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6757888-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssarn0o77h8n6k7tcautgeeq1il2ldi-v20260901", pod="router-65469f6bd6-n84vc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssarn0o77h8n6k7tcautgeeq1il2ldi-v20260901/router-65469f6bd6-n84vc has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T20:42:08Z
  Ended:       2026-09-16T20:53:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6757888-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssari2k27ic1890mj0nf8obfposs9em-j6e9n3v1q9n1f1h", pod="router-65684fdd94-xqlbm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssari2k27ic1890mj0nf8obfposs9em-j6e9n3v1q9n1f1h/router-65684fdd94-xqlbm has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:03:20Z
  Ended:       2026-09-16T21:20:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7138432-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-74fl9", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-74fl9 (namespace arobit) on cluster ci01-j7138432-mgmt-2 has averaged 149.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:07:21Z
  Ended:       2026-09-16T21:20:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7138432-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-58d785c466-8b8jc", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-58d785c466-8b8jc (namespace mgmt-agent) on cluster ci01-j7138432-mgmt-2 has averaged 165.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0xc000f8a120>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-8prbhvsvlkq9" to finish creating: GET https://rp.j9922816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b148abb4-8011-43a5-acf9-7d2f1f124b53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b148abb4-8011-43a5-acf9-7d2f1f124b53",
      "name": "b148abb4-8011-43a5-acf9-7d2f1f124b53",
      "status": "Failed",
      "startTime": "2026-09-16T20:56:50.517658972Z",
      "endTime": "2026-09-16T21:25:54.975318736Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0xc000f8a120>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-8prbhvsvlkq9" to finish creating: GET https://rp.j9922816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b148abb4-8011-43a5-acf9-7d2f1f124b53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b148abb4-8011-43a5-acf9-7d2f1f124b53",
      "name": "b148abb4-8011-43a5-acf9-7d2f1f124b53",
      "status": "Failed",
      "startTime": "2026-09-16T20:56:50.517658972Z",
      "endTime": "2026-09-16T21:25:54.975318736Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:58:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9922816-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8kkxq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8kkxq (namespace arobit) on cluster ci01-j9922816-svc has averaged 125.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:18:46Z
  Ended:       2026-09-16T21:43:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9922816-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6679f7776c-8zsx8", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6679f7776c-8zsx8 (namespace aro-hcp) on cluster ci01-j9922816-svc has averaged 280.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:18:46Z
  Ended:       2026-09-16T21:43:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9922816-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-f87cdfc7-d2vpm", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-f87cdfc7-d2vpm (namespace clusters-service) on cluster ci01-j9922816-svc has averaged 555.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T21:18:46Z
  Ended:       2026-09-16T21:43:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9922816-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-f87cdfc7-kxvdm", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-f87cdfc7-kxvdm (namespace clusters-service) on cluster ci01-j9922816-svc has averaged 181.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T21:18:46Z
  Ended:       2026-09-16T21:43:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9922816-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-f87cdfc7-b8z5c", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-f87cdfc7-b8z5c (namespace clusters-service) on cluster ci01-j9922816-svc has averaged 221.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T21:22:47Z
  Ended:       2026-09-16T21:43:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9922816-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8kkxq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8kkxq (namespace arobit) on cluster ci01-j9922816-svc has averaged 140.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-lzpm5g" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d707c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-lzpm5g: failed waiting for cluster="cp-ystream-upgrade-4-22-lzpm5g" in resourcegroup="rg-cp-ystream-upgrade-4-22-v5f9s54xdgwx" to finish creating: GET https://rp.j9719424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05",
      "name": "cff0006a-d905-424f-9efd-d8db75f2fd05",
      "status": "Failed",
      "startTime": "2026-09-16T20:49:09.853286386Z",
      "endTime": "2026-09-16T21:18:11.008528958Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 13m4s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-lzpm5g" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d707c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-lzpm5g: failed waiting for cluster="cp-ystream-upgrade-4-22-lzpm5g" in resourcegroup="rg-cp-ystream-upgrade-4-22-v5f9s54xdgwx" to finish creating: GET https://rp.j9719424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05",
      "name": "cff0006a-d905-424f-9efd-d8db75f2fd05",
      "status": "Failed",
      "startTime": "2026-09-16T20:49:09.853286386Z",
      "endTime": "2026-09-16T21:18:11.008528958Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 13m4s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:23:50Z
  Ended:       2026-09-16T21:31:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9719424-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-776c4f7db4-4kp88", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-776c4f7db4-4kp88 (namespace clusters-service) on cluster ci01-j9719424-svc has averaged 230.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:23:50Z
  Ended:       2026-09-16T21:31:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9719424-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-nj86k", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-nj86k (namespace arobit) on cluster ci01-j9719424-svc has averaged 155% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T21:23:50Z
  Ended:       2026-09-16T21:31:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9719424-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-bc48745b7-bx9p9", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-bc48745b7-bx9p9 (namespace aro-hcp) on cluster ci01-j9719424-svc has averaged 355.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T21:23:50Z
  Ended:       2026-09-16T21:39:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9719424-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-776c4f7db4-x52fn", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-776c4f7db4-x52fn (namespace clusters-service) on cluster ci01-j9719424-svc has averaged 236.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T21:23:50Z
  Ended:       2026-09-16T21:32:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9719424-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-776c4f7db4-8kcrd", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-776c4f7db4-8kcrd (namespace clusters-service) on cluster ci01-j9719424-svc has averaged 678% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:08:14Z
  Ended:       2026-09-16T21:17:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9709696-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-657b9b568b-npr4g", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-657b9b568b-npr4g (namespace clusters-service) on cluster ci01-j9709696-svc has averaged 301.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:08:14Z
  Ended:       2026-09-16T21:17:16Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9709696-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7d88d477f9-nx2ql", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7d88d477f9-nx2ql (namespace mgmt-agent) on cluster ci01-j9709696-mgmt-2 has averaged 137.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T21:08:15Z
  Ended:       2026-09-16T21:19:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9709696-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-f2zcs", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-f2zcs (namespace arobit) on cluster ci01-j9709696-mgmt-1 has averaged 156% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T20:58:29Z
  Ended:       2026-09-16T21:04:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3887360-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-n8wcv", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-n8wcv (namespace arobit) on cluster ci01-j3887360-mgmt-1 has averaged 210.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00108ec60>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-nzxcxjp96tgd" to finish creating: GET https://rp.j0195072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/68141ca8-41af-45e2-91cd-20fc07c3a848
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/68141ca8-41af-45e2-91cd-20fc07c3a848",
      "name": "68141ca8-41af-45e2-91cd-20fc07c3a848",
      "status": "Failed",
      "startTime": "2026-09-16T19:52:50.957849824Z",
      "endTime": "2026-09-16T20:21:57.186960246Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m31s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00108ec60>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-nzxcxjp96tgd" to finish creating: GET https://rp.j0195072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/68141ca8-41af-45e2-91cd-20fc07c3a848
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/68141ca8-41af-45e2-91cd-20fc07c3a848",
      "name": "68141ca8-41af-45e2-91cd-20fc07c3a848",
      "status": "Failed",
      "startTime": "2026-09-16T19:52:50.957849824Z",
      "endTime": "2026-09-16T20:21:57.186960246Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m31s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T20:25:13Z
  Ended:       2026-09-16T20:35:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0195072-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7fb6485579-nqhtj", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7fb6485579-nqhtj (namespace clusters-service) on cluster ci01-j0195072-svc has averaged 797.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T20:25:13Z
  Ended:       2026-09-16T20:35:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0195072-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8zsgj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8zsgj (namespace arobit) on cluster ci01-j0195072-svc has averaged 278.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T20:26:13Z
  Ended:       2026-09-16T20:34:14Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0195072-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kdkfn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-kdkfn (namespace arobit) on cluster ci01-j0195072-mgmt-2 has averaged 148.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 8 time(s) (1 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T19:54:25Z
  Ended:       2026-09-16T20:12:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7007872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssa6eeifmmv30pii215int1jih1qnml-g2b7m9d9z8p1d0i", pod="router-577b4dc487-wqbd2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssa6eeifmmv30pii215int1jih1qnml-g2b7m9d9z8p1d0i/router-577b4dc487-wqbd2 has been in a non-ready state for longer than 5 minutes.

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T20:16:48Z
  Ended:       2026-09-16T20:24:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7007872-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-qljsw", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-qljsw (namespace arobit) on cluster ci01-j7007872-mgmt-2 has averaged 177.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T20:16:48Z
  Ended:       2026-09-16T20:25:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7007872-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-77765956f4-qcl2v", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-77765956f4-qcl2v (namespace clusters-service) on cluster ci01-j7007872-svc has averaged 253.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T20:16:48Z
  Ended:       2026-09-16T20:32:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7007872-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-jbxwj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-jbxwj (namespace arobit) on cluster ci01-j7007872-svc has averaged 267% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T20:16:48Z
  Ended:       2026-09-16T20:32:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7007872-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-77765956f4-9tcfp", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-77765956f4-9tcfp (namespace clusters-service) on cluster ci01-j7007872-svc has averaged 842.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T20:16:48Z
  Ended:       2026-09-16T20:33:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7007872-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gjkhq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gjkhq (namespace arobit) on cluster ci01-j7007872-mgmt-2 has averaged 187.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T20:16:48Z
  Ended:       2026-09-16T20:32:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7007872-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-f8fcf57b5-bc6mp", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-f8fcf57b5-bc6mp (namespace aro-hcp) on cluster ci01-j7007872-svc has averaged 408.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-16T20:16:48Z
  Ended:       2026-09-16T20:30:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7007872-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-74f949fc76-tvpjx", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-74f949fc76-tvpjx (namespace mgmt-agent) on cluster ci01-j7007872-mgmt-2 has averaged 191% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000a2c540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-86c2sh" in resourcegroup="rg-np-dg-minor-86c2sh-42qzgk6zdhqx" to finish creating: GET https://rp.j4654720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0",
      "name": "b168b436-e53c-4032-86e5-822b4fbf8df0",
      "status": "Failed",
      "startTime": "2026-09-16T19:54:19.817293417Z",
      "endTime": "2026-09-16T19:54:53.786912449Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000a2c540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-86c2sh" in resourcegroup="rg-np-dg-minor-86c2sh-42qzgk6zdhqx" to finish creating: GET https://rp.j4654720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0",
      "name": "b168b436-e53c-4032-86e5-822b4fbf8df0",
      "status": "Failed",
      "startTime": "2026-09-16T19:54:19.817293417Z",
      "endTime": "2026-09-16T19:54:53.786912449Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T20:17:53Z
  Ended:       2026-09-16T20:29:53Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4654720-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5f6fb8bd86-rq9t9", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5f6fb8bd86-rq9t9 (namespace mgmt-agent) on cluster ci01-j4654720-mgmt-1 has averaged 182% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T20:17:53Z
  Ended:       2026-09-16T20:36:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4654720-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-59db946469-cmfdh", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-59db946469-cmfdh (namespace clusters-service) on cluster ci01-j4654720-svc has averaged 244% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T20:17:53Z
  Ended:       2026-09-16T20:29:53Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4654720-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-wwfl9", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-wwfl9 (namespace arobit) on cluster ci01-j4654720-mgmt-1 has averaged 232.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.000s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc0007c5690>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.000s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc0007c5690>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

alert fired 8 time(s) (2 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T19:51:28Z
  Ended:       2026-09-16T20:12:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssa495v0fcuam12oc6m4b88gdkl9b05-l2h4f1u2r7a1i0b", pod="router-85855657b9-6nz7v", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssa495v0fcuam12oc6m4b88gdkl9b05-l2h4f1u2r7a1i0b/router-85855657b9-6nz7v has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T19:51:28Z
  Ended:       2026-09-16T20:03:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssa4occ93fjcinhaca51fjlpiva3rpe-private-ingress", pod="router-68ff4878d4-n82h2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssa4occ93fjcinhaca51fjlpiva3rpe-private-ingress/router-68ff4878d4-n82h2 has been in a non-ready state for longer than 5 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T20:11:42Z
  Ended:       2026-09-16T20:23:44Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j1475328-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-wphcl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-wphcl (namespace arobit) on cluster ci01-j1475328-mgmt-2 has averaged 210.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T20:11:42Z
  Ended:       2026-09-16T20:24:43Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j1475328-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5cbb7b949c-mbxpl", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5cbb7b949c-mbxpl (namespace mgmt-agent) on cluster ci01-j1475328-mgmt-2 has averaged 198.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T20:12:45Z
  Ended:       2026-09-16T20:29:43Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j1475328-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-xrf59", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-xrf59 (namespace arobit) on cluster ci01-j1475328-mgmt-1 has averaged 140.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T20:14:43Z
  Ended:       2026-09-16T20:26:44Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j1475328-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-nzczq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-nzczq (namespace arobit) on cluster ci01-j1475328-mgmt-2 has averaged 139.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00091cf60>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-cdklhkjpj64p" to finish creating: GET https://rp.j2478080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da51fff3-7c72-4b61-92a1-b6c1ea3a06a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da51fff3-7c72-4b61-92a1-b6c1ea3a06a4",
      "name": "da51fff3-7c72-4b61-92a1-b6c1ea3a06a4",
      "status": "Failed",
      "startTime": "2026-09-16T19:31:13.552925362Z",
      "endTime": "2026-09-16T20:00:19.71689726Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m45s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00091cf60>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-cdklhkjpj64p" to finish creating: GET https://rp.j2478080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da51fff3-7c72-4b61-92a1-b6c1ea3a06a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da51fff3-7c72-4b61-92a1-b6c1ea3a06a4",
      "name": "da51fff3-7c72-4b61-92a1-b6c1ea3a06a4",
      "status": "Failed",
      "startTime": "2026-09-16T19:31:13.552925362Z",
      "endTime": "2026-09-16T20:00:19.71689726Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m45s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc00199ac80>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:35223->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc00199ac80>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:35223->172.30.0.10:53: read: connection refused
    ...

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T20:04:07Z
  Ended:       2026-09-16T20:16:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2478080-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7bb74f5747-292f6", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7bb74f5747-292f6 (namespace mgmt-agent) on cluster ci01-j2478080-mgmt-1 has averaged 172.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T20:04:07Z
  Ended:       2026-09-16T20:26:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2478080-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-56db9df8f4-5fz7b", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-56db9df8f4-5fz7b (namespace aro-hcp) on cluster ci01-j2478080-svc has averaged 321.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T20:04:07Z
  Ended:       2026-09-16T20:14:08Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2478080-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-d5fts", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-d5fts (namespace arobit) on cluster ci01-j2478080-mgmt-2 has averaged 154.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T20:04:08Z
  Ended:       2026-09-16T20:26:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2478080-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fd28t", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fd28t (namespace arobit) on cluster ci01-j2478080-svc has averaged 203.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T20:04:08Z
  Ended:       2026-09-16T20:26:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2478080-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5d449f989c-sx24g", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5d449f989c-sx24g (namespace clusters-service) on cluster ci01-j2478080-svc has averaged 663.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T20:04:08Z
  Ended:       2026-09-16T20:14:09Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2478080-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6fb29", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6fb29 (namespace arobit) on cluster ci01-j2478080-mgmt-1 has averaged 148.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-16T20:04:08Z
  Ended:       2026-09-16T20:16:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2478080-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7bb74f5747-7zxzr", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7bb74f5747-7zxzr (namespace mgmt-agent) on cluster ci01-j2478080-mgmt-2 has averaged 181.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000db98e0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-5kbxx5fn4nnf" to finish creating: GET https://rp.j2783232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "name": "ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "status": "Failed",
      "startTime": "2026-09-16T19:26:24.241613754Z",
      "endTime": "2026-09-16T19:55:27.054305259Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000db98e0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-5kbxx5fn4nnf" to finish creating: GET https://rp.j2783232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "name": "ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "status": "Failed",
      "startTime": "2026-09-16T19:26:24.241613754Z",
      "endTime": "2026-09-16T19:55:27.054305259Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T19:58:36Z
  Ended:       2026-09-16T20:12:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2783232-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-99f86f48f-z26v6", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-99f86f48f-z26v6 (namespace mgmt-agent) on cluster ci01-j2783232-mgmt-2 has averaged 219.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T19:58:36Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2783232-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-76d699cfb8-xkqc4", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-76d699cfb8-xkqc4 (namespace clusters-service) on cluster ci01-j2783232-svc has averaged 915.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T19:58:36Z
  Ended:       2026-09-16T20:12:38Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2783232-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gjlth", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gjlth (namespace arobit) on cluster ci01-j2783232-mgmt-2 has averaged 196.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T19:59:38Z
  Ended:       2026-09-16T20:07:36Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2783232-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fm5qj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fm5qj (namespace arobit) on cluster ci01-j2783232-mgmt-1 has averaged 137.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.100s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001341f60>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.100s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001341f60>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T19:21:11Z
  Ended:       2026-09-16T19:31:08Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2121088-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rxj44", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rxj44 (namespace arobit) on cluster ci01-j2121088-svc has averaged 355.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T19:21:11Z
  Ended:       2026-09-16T19:31:08Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2121088-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6b56bfd454-hdslm", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6b56bfd454-hdslm (namespace aro-hcp) on cluster ci01-j2121088-svc has averaged 552.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T19:21:11Z
  Ended:       2026-09-16T19:31:08Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2121088-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-747fb55f57-d9rdv", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-747fb55f57-d9rdv (namespace clusters-service) on cluster ci01-j2121088-svc has averaged 960.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T19:23:13Z
  Ended:       2026-09-16T19:33:11Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2121088-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-67b9d68855-jzclk", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-67b9d68855-jzclk (namespace mgmt-agent) on cluster ci01-j2121088-mgmt-1 has averaged 150.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T19:23:13Z
  Ended:       2026-09-16T19:38:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2121088-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8lsww", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8lsww (namespace arobit) on cluster ci01-j2121088-mgmt-1 has averaged 158.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T19:40:11Z
  Ended:       2026-09-16T19:46:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j2121088-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rxj44", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rxj44 (namespace arobit) on cluster ci01-j2121088-svc has averaged 126.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

time=2026-09-16T18:22:02.494Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-09-16T18:22:51.675Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/1df7c411-d9ac-443d-ac5d-b95681f8419b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/1df7c411-d9ac-443d-ac5d-b95681f8419b\",\n  \"name\": \"1df7c411-d9ac-443d-ac5d-b95681f8419b\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'd478c792-be07-45be-a743-97b4e94abf15'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5982976/providers/Microsoft.Resources/deployments/nbyngfosfawbsx5vtxb2uo6lmfim2rbaopp3emm2zwlmm-260916186gla8\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[0].Expression, Attempted value: null, Error: 1:237: parse error: unknown escape sequence U+002E '.' Activity ID: 95e8cbf4-5734-47ec-859e-d6ed81e0056f.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002420258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-hgxdxgb7d5pf" to finish creating: GET https://rp.j3912704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "name": "bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "status": "Failed",
      "startTime": "2026-09-16T18:52:16.312207682Z",
      "endTime": "2026-09-16T18:52:49.366400193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002420258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-hgxdxgb7d5pf" to finish creating: GET https://rp.j3912704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "name": "bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "status": "Failed",
      "startTime": "2026-09-16T18:52:16.312207682Z",
      "endTime": "2026-09-16T18:52:49.366400193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T19:19:40Z
  Ended:       2026-09-16T19:27:40Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3912704-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5cd45c48f-qb4qc", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5cd45c48f-qb4qc (namespace clusters-service) on cluster ci01-j3912704-svc has averaged 808.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T19:19:40Z
  Ended:       2026-09-16T19:27:40Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3912704-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6f7fcd76b-7dnrd", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6f7fcd76b-7dnrd (namespace aro-hcp) on cluster ci01-j3912704-svc has averaged 423.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T19:19:41Z
  Ended:       2026-09-16T19:27:40Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3912704-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-mv6rx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-mv6rx (namespace arobit) on cluster ci01-j3912704-svc has averaged 236.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T19:19:41Z
  Ended:       2026-09-16T19:26:39Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3912704-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5cd45c48f-x4czc", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5cd45c48f-x4czc (namespace clusters-service) on cluster ci01-j3912704-svc has averaged 243.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T19:14:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j5430016-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-tcbmj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-tcbmj (namespace arobit) on cluster ci01-j5430016-svc has averaged 124.5% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 14 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T18:35:14Z
  Ended:       2026-09-16T19:14:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7767bb5cf4-9nfxx", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7767bb5cf4-9nfxx (namespace clusters-service) on cluster ci01-j5430016-svc has averaged 175.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T18:35:15Z
  Ended:       2026-09-16T18:57:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7767bb5cf4-9smcf", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7767bb5cf4-9smcf (namespace clusters-service) on cluster ci01-j5430016-svc has averaged 175.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T18:35:15Z
  Ended:       2026-09-16T19:21:18Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-8598b596fb-8scn6", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-8598b596fb-8scn6 (namespace aro-hcp) on cluster ci01-j5430016-svc has averaged 335.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T18:35:15Z
  Ended:       2026-09-16T19:18:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-tcbmj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-tcbmj (namespace arobit) on cluster ci01-j5430016-svc has averaged 207.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T18:35:15Z
  Ended:       2026-09-16T19:29:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7767bb5cf4-mz556", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7767bb5cf4-mz556 (namespace clusters-service) on cluster ci01-j5430016-svc has averaged 547.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T18:46:14Z
  Ended:       2026-09-16T18:59:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-xwc94", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-xwc94 (namespace arobit) on cluster ci01-j5430016-mgmt-1 has averaged 142% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-16T18:46:14Z
  Ended:       2026-09-16T18:59:15Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-8fc8c5486-9xpzk", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-8fc8c5486-9xpzk (namespace mgmt-agent) on cluster ci01-j5430016-mgmt-1 has averaged 189.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-16T18:46:14Z
  Ended:       2026-09-16T19:02:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-tmx55", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-tmx55 (namespace arobit) on cluster ci01-j5430016-mgmt-1 has averaged 133.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-16T18:47:15Z
  Ended:       2026-09-16T18:54:16Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-g6qvv", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-g6qvv (namespace arobit) on cluster ci01-j5430016-mgmt-2 has averaged 146.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-16T18:47:15Z
  Ended:       2026-09-16T19:02:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-whfrb", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-whfrb (namespace arobit) on cluster ci01-j5430016-mgmt-2 has averaged 132.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Resolved
  Started:     2026-09-16T18:47:15Z
  Ended:       2026-09-16T19:04:14Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-lzct4", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-lzct4 (namespace arobit) on cluster ci01-j5430016-mgmt-1 has averaged 137.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-16T18:48:15Z
  Ended:       2026-09-16T18:54:16Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-8fc8c5486-klh5t", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-8fc8c5486-klh5t (namespace mgmt-agent) on cluster ci01-j5430016-mgmt-2 has averaged 159.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 13:
  State:       Resolved
  Started:     2026-09-16T18:52:14Z
  Ended:       2026-09-16T19:03:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kqxnt", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-kqxnt (namespace arobit) on cluster ci01-j5430016-mgmt-1 has averaged 136.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 14:
  State:       Resolved
  Started:     2026-09-16T18:54:16Z
  Ended:       2026-09-16T19:03:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5430016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-ncwrg", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-ncwrg (namespace arobit) on cluster ci01-j5430016-mgmt-1 has averaged 129.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

time=2026-09-16T17:32:16.355Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T17:32:17.069Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T17:32:18.139Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=2220b5a771141f6ca7f14ed4462ae3b3d70e20f12f11d2768e401b8114aa657f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7836672-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2220b5a771141f6ca7f14ed4462ae3b3d70e20f12f11d2768e401b8114aa657f
time=2026-09-16T18:02:16.065Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-59s8j", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-59s8j has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-9zlkb", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-9zlkb has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-t8hdl", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-t8hdl has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-49tpv", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-49tpv has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:55:54Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j7836672-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j7836672-svc has been unreachable for 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:55:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7836672/providers/microsoft.monitor/accounts/services-j7836672", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-zdqrc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:55:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7836672/providers/microsoft.monitor/accounts/services-j7836672", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-zdqrc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000bb5820>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xxwwr4bjs848" to finish creating: GET https://rp.j8426240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "name": "13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "status": "Failed",
      "startTime": "2026-09-16T18:00:53.783577799Z",
      "endTime": "2026-09-16T18:19:53.25990245Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000bb5820>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xxwwr4bjs848" to finish creating: GET https://rp.j8426240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "name": "13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "status": "Failed",
      "startTime": "2026-09-16T18:00:53.783577799Z",
      "endTime": "2026-09-16T18:19:53.25990245Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--prf7fs699p2j/cluster-hs-pre-shc7r5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008a9140>: 
    failed to create HCP cluster cluster-hs-pre-shc7r5: failed waiting for cluster="cluster-hs-pre-shc7r5" in resourcegroup="rg-hs-pre--prf7fs699p2j" to finish creating: GET https://rp.j8426240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74928ae1-e11f-4f60-a3b5-4c81b61651aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74928ae1-e11f-4f60-a3b5-4c81b61651aa",
      "name": "74928ae1-e11f-4f60-a3b5-4c81b61651aa",
      "status": "Failed",
      "startTime": "2026-09-16T17:46:42.149025687Z",
      "endTime": "2026-09-16T18:15:43.737568191Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--prf7fs699p2j/cluster-hs-pre-shc7r5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008a9140>: 
    failed to create HCP cluster cluster-hs-pre-shc7r5: failed waiting for cluster="cluster-hs-pre-shc7r5" in resourcegroup="rg-hs-pre--prf7fs699p2j" to finish creating: GET https://rp.j8426240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74928ae1-e11f-4f60-a3b5-4c81b61651aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74928ae1-e11f-4f60-a3b5-4c81b61651aa",
      "name": "74928ae1-e11f-4f60-a3b5-4c81b61651aa",
      "status": "Failed",
      "startTime": "2026-09-16T17:46:42.149025687Z",
      "endTime": "2026-09-16T18:15:43.737568191Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m6s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.001s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.001s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T18:20:28Z
  Ended:       2026-09-16T18:28:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8426240-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6r74z", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6r74z (namespace arobit) on cluster ci01-j8426240-mgmt-2 has averaged 162.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T18:20:28Z
  Ended:       2026-09-16T18:32:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8426240-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6c87p", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6c87p (namespace arobit) on cluster ci01-j8426240-svc has averaged 178.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T18:20:28Z
  Ended:       2026-09-16T18:32:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8426240-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-56d8b8b5b4-b5s2q", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-56d8b8b5b4-b5s2q (namespace clusters-service) on cluster ci01-j8426240-svc has averaged 236.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:58:32Z
  Ended:       2026-09-16T18:19:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8426240-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss8g0v48bf4djgpg9nc4h091itrlbc2-m3n4m9t6z0r3g6f", pod="router-689655689c-bsfzz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss8g0v48bf4djgpg9nc4h091itrlbc2-m3n4m9t6z0r3g6f/router-689655689c-bsfzz has been in a non-ready state for longer than 5 minutes.

time=2026-09-16T17:10:05.170Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-09-16T17:11:24.952Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/aeca7e8d-50c1-47d0-a644-b291a00162ea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/aeca7e8d-50c1-47d0-a644-b291a00162ea\",\n  \"name\": \"aeca7e8d-50c1-47d0-a644-b291a00162ea\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'df67a523-14d6-42a2-9636-444dbb7d2763'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2389376/providers/Microsoft.Resources/deployments/nbyngfosfawbsx5vtxb2uo6lmfim2rbaopp3emm2zwlmm-260916172yp3z\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[0].Expression, Attempted value: null, Error: 1:237: parse error: unknown escape sequence U+002E '.' Activity ID: 91890f9b-76b2-4791-9c30-927a7253a007.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vlzgbk with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc0006bee60>: 
    failed to create HCP cluster np-version-upgrade-cluster-vlzgbk: failed waiting for cluster="np-version-upgrade-cluster-vlzgbk" in resourcegroup="rg-np-version-upgrade-vlzgbk-6gtpz2tlsds2" to finish creating: GET https://rp.j2846464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691",
      "name": "83de4d68-f488-460e-8446-69209da0d691",
      "status": "Failed",
      "startTime": "2026-09-16T17:51:23.682356624Z",
      "endTime": "2026-09-16T18:20:27.523942323Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vlzgbk with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc0006bee60>: 
    failed to create HCP cluster np-version-upgrade-cluster-vlzgbk: failed waiting for cluster="np-version-upgrade-cluster-vlzgbk" in resourcegroup="rg-np-version-upgrade-vlzgbk-6gtpz2tlsds2" to finish creating: GET https://rp.j2846464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691",
      "name": "83de4d68-f488-460e-8446-69209da0d691",
      "status": "Failed",
      "startTime": "2026-09-16T17:51:23.682356624Z",
      "endTime": "2026-09-16T18:20:27.523942323Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:407]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002e66048>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-znw56.apps.aro.agg-cluster.fd32.j2846464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-znw56.apps.aro.agg-cluster.fd32.j2846464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-znw56.apps.aro.agg-cluster.fd32.j2846464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:407]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002e66048>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-znw56.apps.aro.agg-cluster.fd32.j2846464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-znw56.apps.aro.agg-cluster.fd32.j2846464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-znw56.apps.aro.agg-cluster.fd32.j2846464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T18:21:59Z
  Ended:       2026-09-16T18:42:00Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2846464-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2846464/providers/microsoft.monitor/accounts/hcps-j2846464", namespace="ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t", pod="arohcp-monitor-kube-state-metrics-574f657d96-fbvzq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T18:21:59Z
  Ended:       2026-09-16T18:29:59Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2846464-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2846464/providers/microsoft.monitor/accounts/hcps-j2846464", namespace="ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t", pod="arohcp-monitor-kube-state-metrics-574f657d96-fbvzq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T18:12:57Z
  Ended:       2026-09-16T18:25:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2846464-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2846464/providers/microsoft.monitor/accounts/hcps-j2846464", namespace="ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6e854687-f7c2-439d-b129-bec4fba1965e"
  Description: Pod ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-16T18:12:57Z
  Ended:       2026-09-16T18:32:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2846464-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2846464/providers/microsoft.monitor/accounts/hcps-j2846464", namespace="ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6e854687-f7c2-439d-b129-bec4fba1965e"
  Description: Pod ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T18:24:09Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2846464-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-f75b695b-zmfgv", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-f75b695b-zmfgv (namespace mgmt-agent) on cluster ci01-j2846464-mgmt-2 has averaged 282.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T18:24:09Z
  Ended:       2026-09-16T18:48:14Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j2846464-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-m5d7l", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-m5d7l (namespace arobit) on cluster ci01-j2846464-mgmt-2 has averaged 287% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c626c0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wrwln5phv4nq" to finish creating: GET https://rp.j9513728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/000400be-8a8d-4f23-afdd-984e12bfed12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/000400be-8a8d-4f23-afdd-984e12bfed12",
      "name": "000400be-8a8d-4f23-afdd-984e12bfed12",
      "status": "Failed",
      "startTime": "2026-09-16T17:18:18.426850262Z",
      "endTime": "2026-09-16T17:47:24.672789171Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c626c0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wrwln5phv4nq" to finish creating: GET https://rp.j9513728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/000400be-8a8d-4f23-afdd-984e12bfed12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/000400be-8a8d-4f23-afdd-984e12bfed12",
      "name": "000400be-8a8d-4f23-afdd-984e12bfed12",
      "status": "Failed",
      "startTime": "2026-09-16T17:18:18.426850262Z",
      "endTime": "2026-09-16T17:47:24.672789171Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-4k9m7kr6rf4h/cluster-candidate-4-21-76dltv should provision
Unexpected error:
    <*fmt.wrapError | 0xc00116e240>: 
    failed to create HCP cluster cluster-candidate-4-21-76dltv: failed waiting for cluster="cluster-candidate-4-21-76dltv" in resourcegroup="rg-candidate-4-21-4k9m7kr6rf4h" to finish creating: GET https://rp.j9513728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d958346-0f65-40cd-b391-5426f09786f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d958346-0f65-40cd-b391-5426f09786f1",
      "name": "7d958346-0f65-40cd-b391-5426f09786f1",
      "status": "Failed",
      "startTime": "2026-09-16T17:18:18.739919439Z",
      "endTime": "2026-09-16T17:47:24.772096699Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m43s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-4k9m7kr6rf4h/cluster-candidate-4-21-76dltv should provision
Unexpected error:
    <*fmt.wrapError | 0xc00116e240>: 
    failed to create HCP cluster cluster-candidate-4-21-76dltv: failed waiting for cluster="cluster-candidate-4-21-76dltv" in resourcegroup="rg-candidate-4-21-4k9m7kr6rf4h" to finish creating: GET https://rp.j9513728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d958346-0f65-40cd-b391-5426f09786f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d958346-0f65-40cd-b391-5426f09786f1",
      "name": "7d958346-0f65-40cd-b391-5426f09786f1",
      "status": "Failed",
      "startTime": "2026-09-16T17:18:18.739919439Z",
      "endTime": "2026-09-16T17:47:24.772096699Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m43s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:52:05Z
  Ended:       2026-09-16T18:06:07Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9513728-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6f85fb4496-67kgn", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6f85fb4496-67kgn (namespace mgmt-agent) on cluster ci01-j9513728-mgmt-1 has averaged 176.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:52:05Z
  Ended:       2026-09-16T18:04:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j9513728-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kpqlb", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-kpqlb (namespace arobit) on cluster ci01-j9513728-mgmt-1 has averaged 156% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:30:40Z
  Ended:       2026-09-16T18:00:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9513728-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss82miun3vj5m88let2469mn6t6p3gi-v20261001", pod="router-5c4f9947bb-vqdfr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss82miun3vj5m88let2469mn6t6p3gi-v20261001/router-5c4f9947bb-vqdfr has been in a non-ready state for longer than 5 minutes.

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:35:12Z
  Ended:       2026-09-16T17:50:10Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4163200-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-j2smn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-j2smn (namespace arobit) on cluster ci01-j4163200-svc has averaged 173.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:36:10Z
  Ended:       2026-09-16T17:50:10Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4163200-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-6fbd676998-lwslc", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-6fbd676998-lwslc (namespace aro-hcp) on cluster ci01-j4163200-svc has averaged 365.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T17:37:10Z
  Ended:       2026-09-16T17:47:11Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4163200-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-6bb57589d5-9gf7j", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-6bb57589d5-9gf7j (namespace clusters-service) on cluster ci01-j4163200-svc has averaged 173.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:37:10Z
  Ended:       2026-09-16T17:50:10Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4163200-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-6bb57589d5-r4swh", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-6bb57589d5-r4swh (namespace clusters-service) on cluster ci01-j4163200-svc has averaged 645.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T17:37:10Z
  Ended:       2026-09-16T17:43:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4163200-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-6bb57589d5-gh9w9", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-6bb57589d5-gh9w9 (namespace clusters-service) on cluster ci01-j4163200-svc has averaged 178% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T17:44:11Z
  Ended:       2026-09-16T17:53:10Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4163200-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-fdff466bc-5kkx2", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-fdff466bc-5kkx2 (namespace mgmt-agent) on cluster ci01-j4163200-mgmt-1 has averaged 150.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-16T17:48:08Z
  Ended:       2026-09-16T17:58:08Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4163200-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kqxdq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-kqxdq (namespace arobit) on cluster ci01-j4163200-mgmt-1 has averaged 140.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:39:57Z
  Ended:       2026-09-16T17:45:56Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0450304-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0450304-svc).

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:44:57Z
  Ended:       2026-09-16T17:54:55Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0450304-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8rhn6", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8rhn6 (namespace arobit) on cluster ci01-j0450304-mgmt-1 has averaged 243.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:44:57Z
  Ended:       2026-09-16T17:56:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0450304-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-84ccc6569-wlml9", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-84ccc6569-wlml9 (namespace aro-hcp) on cluster ci01-j0450304-svc has averaged 450.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T17:44:57Z
  Ended:       2026-09-16T17:56:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0450304-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-65d5c9b4f7-r55zf", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-65d5c9b4f7-r55zf (namespace clusters-service) on cluster ci01-j0450304-svc has averaged 891% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:44:57Z
  Ended:       2026-09-16T17:51:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0450304-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-trvfl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-trvfl (namespace arobit) on cluster ci01-j0450304-mgmt-2 has averaged 148.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T17:44:57Z
  Ended:       2026-09-16T17:56:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0450304-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-56f9n", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-56f9n (namespace arobit) on cluster ci01-j0450304-svc has averaged 264.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T17:44:57Z
  Ended:       2026-09-16T17:51:56Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0450304-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-787f7cfb65-zfhwc", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-787f7cfb65-zfhwc (namespace mgmt-agent) on cluster ci01-j0450304-mgmt-2 has averaged 166.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:28:09Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8736512-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5d75778c4b-xzbtr", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5d75778c4b-xzbtr (namespace clusters-service) on cluster ci01-j8736512-svc has averaged 655.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:28:09Z
  Ended:       2026-09-16T17:40:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8736512-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-5d75778c4b-hc6kg", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-5d75778c4b-hc6kg (namespace clusters-service) on cluster ci01-j8736512-svc has averaged 197.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T17:28:09Z
  Ended:       2026-09-16T17:35:11Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8736512-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-6777c455fd-5fq84", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-6777c455fd-5fq84 (namespace mgmt-agent) on cluster ci01-j8736512-mgmt-1 has averaged 152.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:37:10Z
  Ended:       2026-09-16T17:44:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8736512-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-cvrzx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-cvrzx (namespace arobit) on cluster ci01-j8736512-mgmt-2 has averaged 127.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:45:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5f5c4b79d-nzh8s", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5f5c4b79d-nzh8s (namespace mgmt-agent) on cluster ci01-j6651904-mgmt-1 has averaged 195.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:45:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-qwq7m", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-qwq7m (namespace arobit) on cluster ci01-j6651904-mgmt-1 has averaged 191.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:27:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-565bdd9568-x62g9", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-565bdd9568-x62g9 (namespace clusters-service) on cluster ci01-j6651904-svc has averaged 720.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:45:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-59bbbb688b-f8hwv", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-59bbbb688b-f8hwv (namespace aro-hcp) on cluster ci01-j6651904-svc has averaged 370.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:44:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-bhgm5", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-bhgm5 (namespace arobit) on cluster ci01-j6651904-mgmt-2 has averaged 175.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:41:35Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-vkvcx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-vkvcx (namespace arobit) on cluster ci01-j6651904-svc has averaged 154.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:45:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-565bdd9568-mf8jt", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-565bdd9568-mf8jt (namespace clusters-service) on cluster ci01-j6651904-svc has averaged 204% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-16T17:31:31Z
  Ended:       2026-09-16T17:47:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6651904-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-565bdd9568-m4fxr", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-565bdd9568-m4fxr (namespace clusters-service) on cluster ci01-j6651904-svc has averaged 262.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-z9mjpt"
Unexpected error:
    <*fmt.wrapError | 0xc0010173e0>: 
    DNS for route host agnhost-e2e-sample-app-6mtvt.apps.aro.d3y6k1b1b2n4e7e.ea63.j6198656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6mtvt.apps.aro.d3y6k1b1b2n4e7e.ea63.j6198656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6mtvt.apps.aro.d3y6k1b1b2n4e7e.ea63.j6198656.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-z9mjpt"
Unexpected error:
    <*fmt.wrapError | 0xc0010173e0>: 
    DNS for route host agnhost-e2e-sample-app-6mtvt.apps.aro.d3y6k1b1b2n4e7e.ea63.j6198656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-6mtvt.apps.aro.d3y6k1b1b2n4e7e.ea63.j6198656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6mtvt.apps.aro.d3y6k1b1b2n4e7e.ea63.j6198656.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-r67qjj"
Unexpected error:
    <*fmt.wrapError | 0xc000814820>: 
    DNS for route host agnhost-e2e-sample-app-zxgt9.apps.aro.j0p9f5n7c5x3h1g.7jfd.j6198656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zxgt9.apps.aro.j0p9f5n7c5x3h1g.7jfd.j6198656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zxgt9.apps.aro.j0p9f5n7c5x3h1g.7jfd.j6198656.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-r67qjj"
Unexpected error:
    <*fmt.wrapError | 0xc000814820>: 
    DNS for route host agnhost-e2e-sample-app-zxgt9.apps.aro.j0p9f5n7c5x3h1g.7jfd.j6198656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zxgt9.apps.aro.j0p9f5n7c5x3h1g.7jfd.j6198656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zxgt9.apps.aro.j0p9f5n7c5x3h1g.7jfd.j6198656.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001ffe8a0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "0IciLMVmLbqimBoU8f5q6QBEpmWjuJNgDzWaCM2v9MM="
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001ffe8a0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "0IciLMVmLbqimBoU8f5q6QBEpmWjuJNgDzWaCM2v9MM="
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:31:16Z
  Ended:       2026-09-16T17:42:17Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0697984-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-7b4b7c85b9-5w7qb", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-7b4b7c85b9-5w7qb (namespace clusters-service) on cluster ci01-j0697984-svc has averaged 629.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-5xvglfvdmmsk/cluster-candidate-4-20-8wb96p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b6a380>: 
    failed to create HCP cluster cluster-candidate-4-20-8wb96p: failed waiting for cluster="cluster-candidate-4-20-8wb96p" in resourcegroup="rg-candidate-4-20-5xvglfvdmmsk" to finish creating: GET https://rp.j3861632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c19faf7-c0c3-42d6-a653-b3ad4df2d164
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c19faf7-c0c3-42d6-a653-b3ad4df2d164",
      "name": "2c19faf7-c0c3-42d6-a653-b3ad4df2d164",
      "status": "Failed",
      "startTime": "2026-09-16T16:55:05.290922061Z",
      "endTime": "2026-09-16T17:24:07.163114799Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 16m23s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-5xvglfvdmmsk/cluster-candidate-4-20-8wb96p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b6a380>: 
    failed to create HCP cluster cluster-candidate-4-20-8wb96p: failed waiting for cluster="cluster-candidate-4-20-8wb96p" in resourcegroup="rg-candidate-4-20-5xvglfvdmmsk" to finish creating: GET https://rp.j3861632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c19faf7-c0c3-42d6-a653-b3ad4df2d164
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c19faf7-c0c3-42d6-a653-b3ad4df2d164",
      "name": "2c19faf7-c0c3-42d6-a653-b3ad4df2d164",
      "status": "Failed",
      "startTime": "2026-09-16T16:55:05.290922061Z",
      "endTime": "2026-09-16T17:24:07.163114799Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 16m23s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000bb8ce0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-r56zm8qhbr8l" to finish creating: GET https://rp.j3861632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/590c2a93-60a1-4c63-b56e-6b6322a7d814
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/590c2a93-60a1-4c63-b56e-6b6322a7d814",
      "name": "590c2a93-60a1-4c63-b56e-6b6322a7d814",
      "status": "Failed",
      "startTime": "2026-09-16T16:55:05.068738086Z",
      "endTime": "2026-09-16T17:24:07.383701828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 16m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000bb8ce0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-r56zm8qhbr8l" to finish creating: GET https://rp.j3861632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/590c2a93-60a1-4c63-b56e-6b6322a7d814
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/590c2a93-60a1-4c63-b56e-6b6322a7d814",
      "name": "590c2a93-60a1-4c63-b56e-6b6322a7d814",
      "status": "Failed",
      "startTime": "2026-09-16T16:55:05.068738086Z",
      "endTime": "2026-09-16T17:24:07.383701828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 16m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:26:31Z
  Ended:       2026-09-16T17:39:34Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3861632-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5f5df44c46-k8k9r", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5f5df44c46-k8k9r (namespace mgmt-agent) on cluster ci01-j3861632-mgmt-1 has averaged 139.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:26:31Z
  Ended:       2026-09-16T17:32:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3861632-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-5f5df44c46-tlbzj", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-5f5df44c46-tlbzj (namespace mgmt-agent) on cluster ci01-j3861632-mgmt-2 has averaged 144.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T17:29:31Z
  Ended:       2026-09-16T17:47:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3861632-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-b9nwv", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-b9nwv (namespace arobit) on cluster ci01-j3861632-mgmt-1 has averaged 142.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:33:32Z
  Ended:       2026-09-16T17:41:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3861632-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-4lnj7", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-4lnj7 (namespace arobit) on cluster ci01-j3861632-mgmt-2 has averaged 141.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:34:29Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3861632-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3861632-svc).

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b4030>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-sk5lk2xqrw7v" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e",
      "name": "234e26dc-5c76-4335-9749-4a1c49aae81e",
      "status": "Failed",
      "startTime": "2026-09-16T17:00:40.136241522Z",
      "endTime": "2026-09-16T17:24:42.83613595Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7is141l1pe3gtmk90b2cej7ltp3gn still exists (deletion dispatched at 2026-09-16T17:00:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b4030>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-sk5lk2xqrw7v" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e",
      "name": "234e26dc-5c76-4335-9749-4a1c49aae81e",
      "status": "Failed",
      "startTime": "2026-09-16T17:00:40.136241522Z",
      "endTime": "2026-09-16T17:24:42.83613595Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7is141l1pe3gtmk90b2cej7ltp3gn still exists (deletion dispatched at 2026-09-16T17:00:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001488048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-2jvpp5kvggw9" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8454bf1b-7bda-408c-958d-23419d4eddce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8454bf1b-7bda-408c-958d-23419d4eddce",
      "name": "8454bf1b-7bda-408c-958d-23419d4eddce",
      "status": "Failed",
      "startTime": "2026-09-16T16:56:41.886166734Z",
      "endTime": "2026-09-16T17:20:42.876661186Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j11fl1lshknqe0flq48ksbqlst7s still exists (deletion dispatched at 2026-09-16T16:56:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001488048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-2jvpp5kvggw9" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8454bf1b-7bda-408c-958d-23419d4eddce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8454bf1b-7bda-408c-958d-23419d4eddce",
      "name": "8454bf1b-7bda-408c-958d-23419d4eddce",
      "status": "Failed",
      "startTime": "2026-09-16T16:56:41.886166734Z",
      "endTime": "2026-09-16T17:20:42.876661186Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j11fl1lshknqe0flq48ksbqlst7s still exists (deletion dispatched at 2026-09-16T16:56:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010ae0d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-gwq76w" in resourcegroup="rg-zstream-upgrade-4-21-kkt25v682kzc" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a1f1fe55-fa55-489a-97de-ae14b6fbabf2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a1f1fe55-fa55-489a-97de-ae14b6fbabf2",
      "name": "a1f1fe55-fa55-489a-97de-ae14b6fbabf2",
      "status": "Failed",
      "startTime": "2026-09-16T17:13:29.218037106Z",
      "endTime": "2026-09-16T17:37:32.760378539Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j1gufforhdfccicslrrj2jqu4vgo still exists (deletion dispatched at 2026-09-16T17:13:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010ae0d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-gwq76w" in resourcegroup="rg-zstream-upgrade-4-21-kkt25v682kzc" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a1f1fe55-fa55-489a-97de-ae14b6fbabf2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a1f1fe55-fa55-489a-97de-ae14b6fbabf2",
      "name": "a1f1fe55-fa55-489a-97de-ae14b6fbabf2",
      "status": "Failed",
      "startTime": "2026-09-16T17:13:29.218037106Z",
      "endTime": "2026-09-16T17:37:32.760378539Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j1gufforhdfccicslrrj2jqu4vgo still exists (deletion dispatched at 2026-09-16T17:13:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001af2090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-ffprrc6mwxtw" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a335987-cce9-48c8-b521-939bf1363ace
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a335987-cce9-48c8-b521-939bf1363ace",
      "name": "8a335987-cce9-48c8-b521-939bf1363ace",
      "status": "Failed",
      "startTime": "2026-09-16T16:57:55.525213606Z",
      "endTime": "2026-09-16T17:22:02.784429606Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j2du92su3qg8tbr84a8d1euhi38t still exists (deletion dispatched at 2026-09-16T16:57:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001af2090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-ffprrc6mwxtw" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a335987-cce9-48c8-b521-939bf1363ace
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8a335987-cce9-48c8-b521-939bf1363ace",
      "name": "8a335987-cce9-48c8-b521-939bf1363ace",
      "status": "Failed",
      "startTime": "2026-09-16T16:57:55.525213606Z",
      "endTime": "2026-09-16T17:22:02.784429606Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j2du92su3qg8tbr84a8d1euhi38t still exists (deletion dispatched at 2026-09-16T16:57:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014240f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xbmtbzvlpclx" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d85e799c-786e-4172-95d1-86693923282f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d85e799c-786e-4172-95d1-86693923282f",
      "name": "d85e799c-786e-4172-95d1-86693923282f",
      "status": "Failed",
      "startTime": "2026-09-16T17:32:57.751739191Z",
      "endTime": "2026-09-16T17:57:03.956555792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j51n0bvbqmvanc2bru5bjg1kuddr still exists (deletion dispatched at 2026-09-16T17:32:58Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014240f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xbmtbzvlpclx" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d85e799c-786e-4172-95d1-86693923282f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d85e799c-786e-4172-95d1-86693923282f",
      "name": "d85e799c-786e-4172-95d1-86693923282f",
      "status": "Failed",
      "startTime": "2026-09-16T17:32:57.751739191Z",
      "endTime": "2026-09-16T17:57:03.956555792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j51n0bvbqmvanc2bru5bjg1kuddr still exists (deletion dispatched at 2026-09-16T17:32:58Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001ab4060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-22-8c8z82" in resourcegroup="rg-zstream-upgrade-4-22-kjwnx2cdrdkq" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02850310-ff4f-4e8b-b9be-3ff8f6b73a30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02850310-ff4f-4e8b-b9be-3ff8f6b73a30",
      "name": "02850310-ff4f-4e8b-b9be-3ff8f6b73a30",
      "status": "Failed",
      "startTime": "2026-09-16T17:13:39.552363722Z",
      "endTime": "2026-09-16T17:37:42.765703073Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7iva927ialh1g5ofg1r8t781qfmn7 still exists (deletion dispatched at 2026-09-16T17:13:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001ab4060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-22-8c8z82" in resourcegroup="rg-zstream-upgrade-4-22-kjwnx2cdrdkq" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02850310-ff4f-4e8b-b9be-3ff8f6b73a30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02850310-ff4f-4e8b-b9be-3ff8f6b73a30",
      "name": "02850310-ff4f-4e8b-b9be-3ff8f6b73a30",
      "status": "Failed",
      "startTime": "2026-09-16T17:13:39.552363722Z",
      "endTime": "2026-09-16T17:37:42.765703073Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7iva927ialh1g5ofg1r8t781qfmn7 still exists (deletion dispatched at 2026-09-16T17:13:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a9c270>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-fqmsfk" in resourcegroup="rg-cp-ystream-upgrade-5-0-lqr7gtcm4srq" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f5ff4f5-7eef-4834-9b6c-c2e1109ad259
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f5ff4f5-7eef-4834-9b6c-c2e1109ad259",
      "name": "3f5ff4f5-7eef-4834-9b6c-c2e1109ad259",
      "status": "Failed",
      "startTime": "2026-09-16T17:13:26.770029811Z",
      "endTime": "2026-09-16T17:37:32.768384818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j3fnpr9m9cqfjaohaiq73k6sjvpi still exists (deletion dispatched at 2026-09-16T17:13:27Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a9c270>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-fqmsfk" in resourcegroup="rg-cp-ystream-upgrade-5-0-lqr7gtcm4srq" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f5ff4f5-7eef-4834-9b6c-c2e1109ad259
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3f5ff4f5-7eef-4834-9b6c-c2e1109ad259",
      "name": "3f5ff4f5-7eef-4834-9b6c-c2e1109ad259",
      "status": "Failed",
      "startTime": "2026-09-16T17:13:26.770029811Z",
      "endTime": "2026-09-16T17:37:32.768384818Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7j3fnpr9m9cqfjaohaiq73k6sjvpi still exists (deletion dispatched at 2026-09-16T17:13:27Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 5 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:00:59Z
  Ended:       2026-09-16T17:19:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4807552-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:00:59Z
  Ended:       2026-09-16T17:19:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4807552-svc", component="backend", name="datadump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T17:00:59Z
  Ended:       2026-09-16T17:20:01Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4807552-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:02:01Z
  Ended:       2026-09-16T17:19:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4807552-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 5:
  State:       Resolved
  Started:     2026-09-16T17:09:02Z
  Ended:       2026-09-16T17:19:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4807552-svc", component="backend", name="triggercontrolplaneupgrade", region="canadacentral", severity="warning"
  Description: Backend controller workqueue triggercontrolplaneupgrade has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:20:46Z
  Ended:       2026-09-16T17:40:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4807552-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5fjfs", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-5fjfs (namespace arobit) on cluster ci01-j4807552-svc has averaged 184.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:20:46Z
  Ended:       2026-09-16T17:46:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4807552-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-86c59d8c49-4xp9s", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-86c59d8c49-4xp9s (namespace clusters-service) on cluster ci01-j4807552-svc has averaged 237.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T17:20:47Z
  Ended:       2026-09-16T17:33:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4807552-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fzgp4", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fzgp4 (namespace arobit) on cluster ci01-j4807552-mgmt-2 has averaged 206.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:20:47Z
  Ended:       2026-09-16T17:33:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4807552-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-8545674775-t5cps", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-8545674775-t5cps (namespace mgmt-agent) on cluster ci01-j4807552-mgmt-2 has averaged 223.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T17:20:47Z
  Ended:       2026-09-16T17:41:46Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4807552-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-86c59d8c49-hskmf", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-86c59d8c49-hskmf (namespace clusters-service) on cluster ci01-j4807552-svc has averaged 213.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T17:20:47Z
  Ended:       2026-09-16T17:54:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4807552-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-64d79bb67-smtrr", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-64d79bb67-smtrr (namespace aro-hcp) on cluster ci01-j4807552-svc has averaged 430.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:54:24Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j4807552-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.89:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4807552/providers/microsoft.monitor/accounts/services-j4807552", namespace="klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn", pod="arohcp-monitor-kube-state-metrics-574f657d96-zvqnn", prometheus="prometheus/prometheus", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn on cluster ci01-j4807552-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:54:24Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j4807552-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.89:8080", job="kube-state-metrics", lease="config-policy-controller", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4807552/providers/microsoft.monitor/accounts/services-j4807552", namespace="klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn", pod="arohcp-monitor-kube-state-metrics-574f657d96-zvqnn", prometheus="prometheus/prometheus", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease config-policy-controller in namespace klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn on cluster ci01-j4807552-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:54:24Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j4807552-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.89:8080", job="kube-state-metrics", lease="governance-policy-framework", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4807552/providers/microsoft.monitor/accounts/services-j4807552", namespace="klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn", pod="arohcp-monitor-kube-state-metrics-574f657d96-zvqnn", prometheus="prometheus/prometheus", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease governance-policy-framework in namespace klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn on cluster ci01-j4807552-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9a460>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ph4gnm2bfn65" to finish creating: GET https://rp.j6091136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea12950c-66bd-4bf3-89b1-519a01fad530
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea12950c-66bd-4bf3-89b1-519a01fad530",
      "name": "ea12950c-66bd-4bf3-89b1-519a01fad530",
      "status": "Failed",
      "startTime": "2026-09-16T16:36:32.246966127Z",
      "endTime": "2026-09-16T17:05:32.533743336Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9a460>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ph4gnm2bfn65" to finish creating: GET https://rp.j6091136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea12950c-66bd-4bf3-89b1-519a01fad530
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea12950c-66bd-4bf3-89b1-519a01fad530",
      "name": "ea12950c-66bd-4bf3-89b1-519a01fad530",
      "status": "Failed",
      "startTime": "2026-09-16T16:36:32.246966127Z",
      "endTime": "2026-09-16T17:05:32.533743336Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:36:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-cwxxr has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T16:36:14Z
  Ended:       2026-09-16T17:19:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-28181016-vmss000000-vxff5", region="canadacentral", severity="warning"
  Description: Pod kube-system/eraser-aks-system-28181016-vmss000000-vxff5 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:36:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-xfmjp", region="canadacentral", severity="warning"
  Description: Pod kube-system/ama-metrics-node-xfmjp has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:11:51Z
  Ended:       2026-09-16T17:29:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/hcps-j6091136", namespace="ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:45:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:45:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:46:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:46:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.

alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="secrets-store", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-cwxxr in namespace kube-system on container secrets-store has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="liveness-probe", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-cwxxr in namespace kube-system on container liveness-probe has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="node-driver-registrar", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-cwxxr in namespace kube-system on container node-driver-registrar has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="prometheus-collector", namespace="kube-system", pod="ama-metrics-node-xfmjp", region="canadacentral", severity="warning"
  Description: pod/ama-metrics-node-xfmjp in namespace kube-system on container prometheus-collector has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="addon-token-adapter", namespace="kube-system", pod="ama-metrics-node-xfmjp", region="canadacentral", severity="warning"
  Description: pod/ama-metrics-node-xfmjp in namespace kube-system on container addon-token-adapter has been in waiting state for longer than 1 hour.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T16:56:51Z
  Ended:       2026-09-16T17:24:50Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/hcps-j6091136", namespace="ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:00:36Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="prometheus", node="aks-system-28181016-vmss000000", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-system-28181016-vmss000000 has been unready for more than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:00:36Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="prometheus", node="aks-system-28181016-vmss000000", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-system-28181016-vmss000000 has been unready for more than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:11:15Z
  Ended:       2026-09-16T17:17:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="klusterlet-2ss7f44euk2o4nqkrp28r72hngjs9t5m", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ss7f44euk2o4nqkrp28r72hngjs9t5m/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:11:36Z
  Ended:       2026-09-16T17:33:35Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6091136-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8565d85cf5-nxd7f", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8565d85cf5-nxd7f (namespace clusters-service) on cluster ci01-j6091136-svc has averaged 261.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T17:11:36Z
  Ended:       2026-09-16T17:21:35Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6091136-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-59b756d996-zw9bn", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-59b756d996-zw9bn (namespace mgmt-agent) on cluster ci01-j6091136-mgmt-1 has averaged 157.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T17:11:36Z
  Ended:       2026-09-16T17:19:36Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6091136-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-fjjcj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-fjjcj (namespace arobit) on cluster ci01-j6091136-mgmt-1 has averaged 150.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-16T17:11:36Z
  Ended:       2026-09-16T17:29:36Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6091136-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8565d85cf5-xznk2", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8565d85cf5-xznk2 (namespace clusters-service) on cluster ci01-j6091136-svc has averaged 235.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T17:11:36Z
  Ended:       2026-09-16T17:29:37Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6091136-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-5b49bd5fc7-4mpg6", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-5b49bd5fc7-4mpg6 (namespace aro-hcp) on cluster ci01-j6091136-svc has averaged 438.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T17:11:36Z
  Ended:       2026-09-16T17:29:37Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j6091136-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gttgk", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gttgk (namespace arobit) on cluster ci01-j6091136-svc has averaged 224.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T16:39:46Z
  Ended:       2026-09-16T17:12:43Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6091136-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6091136-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T16:36:22Z
  Ended:       2026-09-16T16:43:22Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j4323840-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-gctwd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-gctwd (namespace arobit) on cluster ci01-j4323840-mgmt-2 has averaged 156.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.010s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001cb64b0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.010s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001cb64b0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T16:34:22Z
  Ended:       2026-09-16T16:42:25Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j8880128-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-856989784f-nd6qv", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-856989784f-nd6qv (namespace clusters-service) on cluster ci01-j8880128-svc has averaged 259.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:144]: Admin credentials did not return 409 conflict error while cluster is deploying
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:144]: Admin credentials did not return 409 conflict error while cluster is deploying

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc0018f0700>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc0018f0700>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

time=2026-09-16T15:02:01.563Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T15:02:02.231Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T15:02:03.235Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=9b458a858b62ced8b861c50221db722ca7b84af5c0247e7866fc1bf068dbe8ac portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7202944-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9b458a858b62ced8b861c50221db722ca7b84af5c0247e7866fc1bf068dbe8ac
time=2026-09-16T15:11:05.045Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/9b458a858b62ced8b861c50221db722ca7b84af5c0247e7866fc1bf068dbe8ac/operationStatuses/08584120359627896142\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/cluster-oqca7h7ulqwlw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7202944-mgmt-2/agentPools/userswft3\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 96, remaining 48 for family StandardEdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7202944-mgmt-2/agentPools/userswft2\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 96, remaining 48 for family StandardEdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7202944-mgmt-2/agentPools/userswft1\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 96, remaining 48 for family StandardEdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000c1a440>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6zdxmb6rpjzk" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/05315bbb-661b-4d11-978b-563d64e9ecf5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/05315bbb-661b-4d11-978b-563d64e9ecf5",
      "name": "05315bbb-661b-4d11-978b-563d64e9ecf5",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:49.128060799Z",
      "endTime": "2026-09-16T15:51:54.335440454Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m47s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000c1a440>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6zdxmb6rpjzk" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/05315bbb-661b-4d11-978b-563d64e9ecf5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/05315bbb-661b-4d11-978b-563d64e9ecf5",
      "name": "05315bbb-661b-4d11-978b-563d64e9ecf5",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:49.128060799Z",
      "endTime": "2026-09-16T15:51:54.335440454Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m47s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000a8ea00>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-67blpgs4n85p" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/41037149-7747-47d4-afb8-5a24484b3554
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/41037149-7747-47d4-afb8-5a24484b3554",
      "name": "41037149-7747-47d4-afb8-5a24484b3554",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:40.908389844Z",
      "endTime": "2026-09-16T15:51:44.402910752Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000a8ea00>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-67blpgs4n85p" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/41037149-7747-47d4-afb8-5a24484b3554
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/41037149-7747-47d4-afb8-5a24484b3554",
      "name": "41037149-7747-47d4-afb8-5a24484b3554",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:40.908389844Z",
      "endTime": "2026-09-16T15:51:44.402910752Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000d7dc40>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-7vbqmtrz9vrj" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a28fe40a-0fe9-4492-945f-7a7bb07d9075
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a28fe40a-0fe9-4492-945f-7a7bb07d9075",
      "name": "a28fe40a-0fe9-4492-945f-7a7bb07d9075",
      "status": "Failed",
      "startTime": "2026-09-16T15:24:46.641065736Z",
      "endTime": "2026-09-16T15:53:54.422419253Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000d7dc40>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-7vbqmtrz9vrj" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a28fe40a-0fe9-4492-945f-7a7bb07d9075
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a28fe40a-0fe9-4492-945f-7a7bb07d9075",
      "name": "a28fe40a-0fe9-4492-945f-7a7bb07d9075",
      "status": "Failed",
      "startTime": "2026-09-16T15:24:46.641065736Z",
      "endTime": "2026-09-16T15:53:54.422419253Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d66760>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6s59tj5f5cwc" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf873d80-2639-4560-9d03-e297040bde4f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf873d80-2639-4560-9d03-e297040bde4f",
      "name": "bf873d80-2639-4560-9d03-e297040bde4f",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:59.281142097Z",
      "endTime": "2026-09-16T15:52:04.370330727Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m17s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d66760>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6s59tj5f5cwc" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf873d80-2639-4560-9d03-e297040bde4f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf873d80-2639-4560-9d03-e297040bde4f",
      "name": "bf873d80-2639-4560-9d03-e297040bde4f",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:59.281142097Z",
      "endTime": "2026-09-16T15:52:04.370330727Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m17s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000efe540>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-62bnx4vg8k7q" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ea6c341-7276-4abb-bbae-9fdf251b0d28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ea6c341-7276-4abb-bbae-9fdf251b0d28",
      "name": "3ea6c341-7276-4abb-bbae-9fdf251b0d28",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:30.840455999Z",
      "endTime": "2026-09-16T15:51:34.353957305Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m54s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000efe540>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-62bnx4vg8k7q" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ea6c341-7276-4abb-bbae-9fdf251b0d28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3ea6c341-7276-4abb-bbae-9fdf251b0d28",
      "name": "3ea6c341-7276-4abb-bbae-9fdf251b0d28",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:30.840455999Z",
      "endTime": "2026-09-16T15:51:34.353957305Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m54s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000d32180>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-bvtz5scvbwbv" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d32da0c1-6650-42fe-a236-8452aa971c7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d32da0c1-6650-42fe-a236-8452aa971c7a",
      "name": "d32da0c1-6650-42fe-a236-8452aa971c7a",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:40.242608551Z",
      "endTime": "2026-09-16T15:51:44.326484397Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m4s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000d32180>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-bvtz5scvbwbv" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d32da0c1-6650-42fe-a236-8452aa971c7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d32da0c1-6650-42fe-a236-8452aa971c7a",
      "name": "d32da0c1-6650-42fe-a236-8452aa971c7a",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:40.242608551Z",
      "endTime": "2026-09-16T15:51:44.326484397Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m4s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000e4c880>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-7vfjq4g5mpkz" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa4772cb-eb8a-477d-b8e0-5d1f516c5a63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa4772cb-eb8a-477d-b8e0-5d1f516c5a63",
      "name": "fa4772cb-eb8a-477d-b8e0-5d1f516c5a63",
      "status": "Failed",
      "startTime": "2026-09-16T15:25:02.43485078Z",
      "endTime": "2026-09-16T15:54:04.537444594Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000e4c880>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-7vfjq4g5mpkz" to finish creating: GET https://rp.j3034240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa4772cb-eb8a-477d-b8e0-5d1f516c5a63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fa4772cb-eb8a-477d-b8e0-5d1f516c5a63",
      "name": "fa4772cb-eb8a-477d-b8e0-5d1f516c5a63",
      "status": "Failed",
      "startTime": "2026-09-16T15:25:02.43485078Z",
      "endTime": "2026-09-16T15:54:04.537444594Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:37:33Z
  Ended:       2026-09-16T16:04:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3034240-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss6ciiqtdo86qknpl3sl69pgldega5p-a3j0h6i8k2j6q8r", pod="router-64fb7b56c-wnsxq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss6ciiqtdo86qknpl3sl69pgldega5p-a3j0h6i8k2j6q8r/router-64fb7b56c-wnsxq has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:444]: failed to create HCP cluster np-skip-minor-xf4ktl at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc000d984e0>: 
    failed to create HCP cluster np-skip-minor-xf4ktl: failed waiting for cluster="np-skip-minor-xf4ktl" in resourcegroup="rg-np-skip-minor-xf4ktl-ndhlhsxtb9bp" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9ddbedd-dd2c-4969-be11-096f362f19a9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9ddbedd-dd2c-4969-be11-096f362f19a9",
      "name": "d9ddbedd-dd2c-4969-be11-096f362f19a9",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:55.657642282Z",
      "endTime": "2026-09-16T15:38:00.339938436Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:444]: failed to create HCP cluster np-skip-minor-xf4ktl at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc000d984e0>: 
    failed to create HCP cluster np-skip-minor-xf4ktl: failed waiting for cluster="np-skip-minor-xf4ktl" in resourcegroup="rg-np-skip-minor-xf4ktl-ndhlhsxtb9bp" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9ddbedd-dd2c-4969-be11-096f362f19a9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d9ddbedd-dd2c-4969-be11-096f362f19a9",
      "name": "d9ddbedd-dd2c-4969-be11-096f362f19a9",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:55.657642282Z",
      "endTime": "2026-09-16T15:38:00.339938436Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc001dd40f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2dq2jrpmm79t" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e014f5fe-d804-4d74-8be2-a041ccc5139a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e014f5fe-d804-4d74-8be2-a041ccc5139a",
      "name": "e014f5fe-d804-4d74-8be2-a041ccc5139a",
      "status": "Failed",
      "startTime": "2026-09-16T15:38:30.175216442Z",
      "endTime": "2026-09-16T16:02:31.265925549Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss65h8jgj0ruglgg8jptfh3v6s7qujv still exists (deletion dispatched at 2026-09-16T15:38:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc001dd40f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2dq2jrpmm79t" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e014f5fe-d804-4d74-8be2-a041ccc5139a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e014f5fe-d804-4d74-8be2-a041ccc5139a",
      "name": "e014f5fe-d804-4d74-8be2-a041ccc5139a",
      "status": "Failed",
      "startTime": "2026-09-16T15:38:30.175216442Z",
      "endTime": "2026-09-16T16:02:31.265925549Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss65h8jgj0ruglgg8jptfh3v6s7qujv still exists (deletion dispatched at 2026-09-16T15:38:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:103]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001f3c120>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sxmn5zhcsvjn" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "name": "a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:36.527287031Z",
      "endTime": "2026-09-16T15:37:40.20476418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:103]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001f3c120>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sxmn5zhcsvjn" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "name": "a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:36.527287031Z",
      "endTime": "2026-09-16T15:37:40.20476418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e96840>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-222ntj7gxqx2" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "name": "774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.983252228Z",
      "endTime": "2026-09-16T15:36:40.328092646Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e96840>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-222ntj7gxqx2" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "name": "774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.983252228Z",
      "endTime": "2026-09-16T15:36:40.328092646Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00105cce0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-l5lpwk8c788b" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75be2192-167a-440e-8345-e0e86c88eab4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75be2192-167a-440e-8345-e0e86c88eab4",
      "name": "75be2192-167a-440e-8345-e0e86c88eab4",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:49.433866246Z",
      "endTime": "2026-09-16T15:36:50.290217783Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00105cce0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-l5lpwk8c788b" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75be2192-167a-440e-8345-e0e86c88eab4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75be2192-167a-440e-8345-e0e86c88eab4",
      "name": "75be2192-167a-440e-8345-e0e86c88eab4",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:49.433866246Z",
      "endTime": "2026-09-16T15:36:50.290217783Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000861120>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-pkbzl9cwqzmf" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823",
      "name": "7bc0c172-a0d6-40dc-821f-14c51cede823",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:59.260634412Z",
      "endTime": "2026-09-16T15:37:00.334559736Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000861120>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-pkbzl9cwqzmf" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823",
      "name": "7bc0c172-a0d6-40dc-821f-14c51cede823",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:59.260634412Z",
      "endTime": "2026-09-16T15:37:00.334559736Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:112]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000baef60>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-r66n4dfd4blw" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e8da50e-214e-46a9-b300-dcbc9025d457
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e8da50e-214e-46a9-b300-dcbc9025d457",
      "name": "5e8da50e-214e-46a9-b300-dcbc9025d457",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:22.677421205Z",
      "endTime": "2026-09-16T15:37:30.284421662Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:112]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000baef60>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-r66n4dfd4blw" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e8da50e-214e-46a9-b300-dcbc9025d457
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5e8da50e-214e-46a9-b300-dcbc9025d457",
      "name": "5e8da50e-214e-46a9-b300-dcbc9025d457",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:22.677421205Z",
      "endTime": "2026-09-16T15:37:30.284421662Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:420]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000e08340>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rjd9q4sr4pnl" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de1336a6-9373-4313-bfd1-74e55afef59f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de1336a6-9373-4313-bfd1-74e55afef59f",
      "name": "de1336a6-9373-4313-bfd1-74e55afef59f",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:25.479355581Z",
      "endTime": "2026-09-16T15:37:30.356691506Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:420]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000e08340>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-rjd9q4sr4pnl" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de1336a6-9373-4313-bfd1-74e55afef59f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de1336a6-9373-4313-bfd1-74e55afef59f",
      "name": "de1336a6-9373-4313-bfd1-74e55afef59f",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:25.479355581Z",
      "endTime": "2026-09-16T15:37:30.356691506Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d8e640>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mldclqf9sr5h" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44",
      "name": "704efcc4-4b42-4704-9ff8-f01712701a44",
      "status": "Failed",
      "startTime": "2026-09-16T15:09:00.206878787Z",
      "endTime": "2026-09-16T15:38:00.28474108Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d8e640>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mldclqf9sr5h" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44",
      "name": "704efcc4-4b42-4704-9ff8-f01712701a44",
      "status": "Failed",
      "startTime": "2026-09-16T15:09:00.206878787Z",
      "endTime": "2026-09-16T15:38:00.28474108Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc000e90390>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-lqz4p67hcxjl" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ebdfc80f-b540-48be-aab0-8a738a247fdb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ebdfc80f-b540-48be-aab0-8a738a247fdb",
      "name": "ebdfc80f-b540-48be-aab0-8a738a247fdb",
      "status": "Failed",
      "startTime": "2026-09-16T15:36:32.444402833Z",
      "endTime": "2026-09-16T16:00:41.296180241Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss6623gm9lp6mh9kjc1ql22lej4dllq still exists (deletion dispatched at 2026-09-16T15:36:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc000e90390>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-lqz4p67hcxjl" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ebdfc80f-b540-48be-aab0-8a738a247fdb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ebdfc80f-b540-48be-aab0-8a738a247fdb",
      "name": "ebdfc80f-b540-48be-aab0-8a738a247fdb",
      "status": "Failed",
      "startTime": "2026-09-16T15:36:32.444402833Z",
      "endTime": "2026-09-16T16:00:41.296180241Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss6623gm9lp6mh9kjc1ql22lej4dllq still exists (deletion dispatched at 2026-09-16T15:36:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc0013de078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-7jq9hm2md6t8" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf6b5b4a-2ab2-4c6d-916c-70bbc5bfcd3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf6b5b4a-2ab2-4c6d-916c-70bbc5bfcd3c",
      "name": "cf6b5b4a-2ab2-4c6d-916c-70bbc5bfcd3c",
      "status": "Failed",
      "startTime": "2026-09-16T15:35:11.305500228Z",
      "endTime": "2026-09-16T15:59:11.284085711Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss6632a8medpq72h75idr9hq6fk0kqj still exists (deletion dispatched at 2026-09-16T15:35:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc0013de078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-7jq9hm2md6t8" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf6b5b4a-2ab2-4c6d-916c-70bbc5bfcd3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf6b5b4a-2ab2-4c6d-916c-70bbc5bfcd3c",
      "name": "cf6b5b4a-2ab2-4c6d-916c-70bbc5bfcd3c",
      "status": "Failed",
      "startTime": "2026-09-16T15:35:11.305500228Z",
      "endTime": "2026-09-16T15:59:11.284085711Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss6632a8medpq72h75idr9hq6fk0kqj still exists (deletion dispatched at 2026-09-16T15:35:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:206]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc002a80740>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-czltd5xxrlvj" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "name": "180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "status": "Failed",
      "startTime": "2026-09-16T15:10:24.910079282Z",
      "endTime": "2026-09-16T15:39:30.319726782Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:206]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc002a80740>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-czltd5xxrlvj" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "name": "180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "status": "Failed",
      "startTime": "2026-09-16T15:10:24.910079282Z",
      "endTime": "2026-09-16T15:39:30.319726782Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:85]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b250e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-rc8rlbszt67v" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/954e120a-f424-45bb-8dc4-e1e41d681776
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/954e120a-f424-45bb-8dc4-e1e41d681776",
      "name": "954e120a-f424-45bb-8dc4-e1e41d681776",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.834280426Z",
      "endTime": "2026-09-16T15:36:40.350051467Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:85]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b250e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-rc8rlbszt67v" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/954e120a-f424-45bb-8dc4-e1e41d681776
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/954e120a-f424-45bb-8dc4-e1e41d681776",
      "name": "954e120a-f424-45bb-8dc4-e1e41d681776",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.834280426Z",
      "endTime": "2026-09-16T15:36:40.350051467Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m14s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:183]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:183]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f48160>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hdp82z6x6stg" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/57c282c3-8a3f-435c-8e84-cc6479b9b24d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/57c282c3-8a3f-435c-8e84-cc6479b9b24d",
      "name": "57c282c3-8a3f-435c-8e84-cc6479b9b24d",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:40.381050368Z",
      "endTime": "2026-09-16T15:36:40.345057819Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 14m43s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f48160>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hdp82z6x6stg" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/57c282c3-8a3f-435c-8e84-cc6479b9b24d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/57c282c3-8a3f-435c-8e84-cc6479b9b24d",
      "name": "57c282c3-8a3f-435c-8e84-cc6479b9b24d",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:40.381050368Z",
      "endTime": "2026-09-16T15:36:40.345057819Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 14m43s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000d24800>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-t46rl8c755dd" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a93fc974-cd28-42a1-85bb-63ca024b7fae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a93fc974-cd28-42a1-85bb-63ca024b7fae",
      "name": "a93fc974-cd28-42a1-85bb-63ca024b7fae",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:54.45273143Z",
      "endTime": "2026-09-16T15:38:00.30841047Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000d24800>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-t46rl8c755dd" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a93fc974-cd28-42a1-85bb-63ca024b7fae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a93fc974-cd28-42a1-85bb-63ca024b7fae",
      "name": "a93fc974-cd28-42a1-85bb-63ca024b7fae",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:54.45273143Z",
      "endTime": "2026-09-16T15:38:00.30841047Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: failed to create HCP cluster "cluster-zstream-5-0-ltfq8d" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00019fbc0>: 
    failed to create HCP cluster cluster-zstream-5-0-ltfq8d: failed waiting for cluster="cluster-zstream-5-0-ltfq8d" in resourcegroup="rg-zstream-upgrade-5-0-b9vtt7qlr7t6" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c61b5bd2-2bcc-415d-a1a0-e9dd864a49f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c61b5bd2-2bcc-415d-a1a0-e9dd864a49f0",
      "name": "c61b5bd2-2bcc-415d-a1a0-e9dd864a49f0",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:39.815926407Z",
      "endTime": "2026-09-16T15:36:40.332553644Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.1 is Partial (want Completed), started 16m33s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: failed to create HCP cluster "cluster-zstream-5-0-ltfq8d" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00019fbc0>: 
    failed to create HCP cluster cluster-zstream-5-0-ltfq8d: failed waiting for cluster="cluster-zstream-5-0-ltfq8d" in resourcegroup="rg-zstream-upgrade-5-0-b9vtt7qlr7t6" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c61b5bd2-2bcc-415d-a1a0-e9dd864a49f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c61b5bd2-2bcc-415d-a1a0-e9dd864a49f0",
      "name": "c61b5bd2-2bcc-415d-a1a0-e9dd864a49f0",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:39.815926407Z",
      "endTime": "2026-09-16T15:36:40.332553644Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.1 is Partial (want Completed), started 16m33s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--7tj5b6r9d28k/cluster-hs-pre-rwztb2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ee48a0>: 
    failed to create HCP cluster cluster-hs-pre-rwztb2: failed waiting for cluster="cluster-hs-pre-rwztb2" in resourcegroup="rg-hs-pre--7tj5b6r9d28k" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d",
      "name": "0015f221-0420-4231-b96b-76ce90032f9d",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.092292569Z",
      "endTime": "2026-09-16T15:36:40.334174064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--7tj5b6r9d28k/cluster-hs-pre-rwztb2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ee48a0>: 
    failed to create HCP cluster cluster-hs-pre-rwztb2: failed waiting for cluster="cluster-hs-pre-rwztb2" in resourcegroup="rg-hs-pre--7tj5b6r9d28k" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d",
      "name": "0015f221-0420-4231-b96b-76ce90032f9d",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.092292569Z",
      "endTime": "2026-09-16T15:36:40.334174064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:111]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0012ca380>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-t5bwqzxttz2x" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7634186-3d3c-4d05-8ca0-3b5b6a83a2ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7634186-3d3c-4d05-8ca0-3b5b6a83a2ab",
      "name": "c7634186-3d3c-4d05-8ca0-3b5b6a83a2ab",
      "status": "Failed",
      "startTime": "2026-09-16T15:40:50.62183993Z",
      "endTime": "2026-09-16T16:09:59.134370699Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 25m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:111]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0012ca380>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-t5bwqzxttz2x" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7634186-3d3c-4d05-8ca0-3b5b6a83a2ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c7634186-3d3c-4d05-8ca0-3b5b6a83a2ab",
      "name": "c7634186-3d3c-4d05-8ca0-3b5b6a83a2ab",
      "status": "Failed",
      "startTime": "2026-09-16T15:40:50.62183993Z",
      "endTime": "2026-09-16T16:09:59.134370699Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 25m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc00228a378>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-rsrv4kf2w46v" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8e49de5-12cd-4e7e-9b28-73dc76953ac6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8e49de5-12cd-4e7e-9b28-73dc76953ac6",
      "name": "e8e49de5-12cd-4e7e-9b28-73dc76953ac6",
      "status": "Failed",
      "startTime": "2026-09-16T15:42:15.675788401Z",
      "endTime": "2026-09-16T16:06:21.284864271Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss6634t6efhs2ajqtn63hq5cv2cmsie still exists (deletion dispatched at 2026-09-16T15:42:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc00228a378>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-rsrv4kf2w46v" to finish deleting: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8e49de5-12cd-4e7e-9b28-73dc76953ac6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e8e49de5-12cd-4e7e-9b28-73dc76953ac6",
      "name": "e8e49de5-12cd-4e7e-9b28-73dc76953ac6",
      "status": "Failed",
      "startTime": "2026-09-16T15:42:15.675788401Z",
      "endTime": "2026-09-16T16:06:21.284864271Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss6634t6efhs2ajqtn63hq5cv2cmsie still exists (deletion dispatched at 2026-09-16T15:42:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:584]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101ca40>: 
    failed to create HCP cluster np-downgrade-ptnt6v: failed waiting for cluster="np-downgrade-ptnt6v" in resourcegroup="rg-np-downgrade-ptnt6v-s6586bctkvvs" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "name": "b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:45.747660708Z",
      "endTime": "2026-09-16T15:36:50.286764481Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:584]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101ca40>: 
    failed to create HCP cluster np-downgrade-ptnt6v: failed waiting for cluster="np-downgrade-ptnt6v" in resourcegroup="rg-np-downgrade-ptnt6v-s6586bctkvvs" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "name": "b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:45.747660708Z",
      "endTime": "2026-09-16T15:36:50.286764481Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ca4420>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-nspddm98jhdn" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a021ad-4d63-43da-b693-4efe761edec6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a021ad-4d63-43da-b693-4efe761edec6",
      "name": "c9a021ad-4d63-43da-b693-4efe761edec6",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:40.588728037Z",
      "endTime": "2026-09-16T15:36:40.298013958Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ca4420>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-nspddm98jhdn" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a021ad-4d63-43da-b693-4efe761edec6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9a021ad-4d63-43da-b693-4efe761edec6",
      "name": "c9a021ad-4d63-43da-b693-4efe761edec6",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:40.588728037Z",
      "endTime": "2026-09-16T15:36:40.298013958Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-f27q54rmwchw/cluster-candidate-4-21-s5z6tn should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f28500>: 
    failed to create HCP cluster cluster-candidate-4-21-s5z6tn: failed waiting for cluster="cluster-candidate-4-21-s5z6tn" in resourcegroup="rg-candidate-4-21-f27q54rmwchw" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/336966d4-bc9c-4965-a943-7c74b3b76a9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/336966d4-bc9c-4965-a943-7c74b3b76a9a",
      "name": "336966d4-bc9c-4965-a943-7c74b3b76a9a",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:40.621415385Z",
      "endTime": "2026-09-16T15:36:40.360524943Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 16m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-f27q54rmwchw/cluster-candidate-4-21-s5z6tn should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f28500>: 
    failed to create HCP cluster cluster-candidate-4-21-s5z6tn: failed waiting for cluster="cluster-candidate-4-21-s5z6tn" in resourcegroup="rg-candidate-4-21-f27q54rmwchw" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/336966d4-bc9c-4965-a943-7c74b3b76a9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/336966d4-bc9c-4965-a943-7c74b3b76a9a",
      "name": "336966d4-bc9c-4965-a943-7c74b3b76a9a",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:40.621415385Z",
      "endTime": "2026-09-16T15:36:40.360524943Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 16m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0009db7a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wngrxcmwz6zh" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901AndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-wngrxcmwz6zh, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0009db7a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wngrxcmwz6zh" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901AndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-wngrxcmwz6zh, error: context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:50:51Z
  Ended:       2026-09-16T15:58:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.65.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7769984/providers/microsoft.monitor/accounts/services-j7769984", namespace="klusterlet-2ss6631b7kcrqufnkep1rdniihr6hk1s", pod="arohcp-monitor-kube-state-metrics-574f657d96-n4qlf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2ss6631b7kcrqufnkep1rdniihr6hk1s/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:30:00Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7769984/providers/microsoft.monitor/accounts/hcps-j7769984", namespace="ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-n4qlf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:30:00Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7769984/providers/microsoft.monitor/accounts/hcps-j7769984", namespace="ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-n4qlf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.

alert fired 7 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:23:56Z
  Ended:       2026-09-16T15:50:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7769984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss65gp6p7cj2egifuj36635qckn05a0-p3q5k4a8x1k8d8w", pod="router-6f94f5d6c4-ccjv2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss65gp6p7cj2egifuj36635qckn05a0-p3q5k4a8x1k8d8w/router-6f94f5d6c4-ccjv2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T15:24:56Z
  Ended:       2026-09-16T15:30:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss65mnnlogu9073uivmq2325bj8tnd2-u0g9g1i5r8s0n6y", pod="router-854fb99ff8-k2fd5", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss65mnnlogu9073uivmq2325bj8tnd2-u0g9g1i5r8s0n6y/router-854fb99ff8-k2fd5 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T15:24:56Z
  Ended:       2026-09-16T15:50:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7769984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss65hn0kjsbu0s5o2aeg7qk5e1ajkjb-v20260901", pod="router-78df7bd99d-57jjt", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss65hn0kjsbu0s5o2aeg7qk5e1ajkjb-v20260901/router-78df7bd99d-57jjt has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T15:24:56Z
  Ended:       2026-09-16T15:30:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss6623gm9lp6mh9kjc1ql22lej4dllq-oidc-wi-cluster", pod="router-d46bb467b-5g27g", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss6623gm9lp6mh9kjc1ql22lej4dllq-oidc-wi-cluster/router-d46bb467b-5g27g has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-16T15:24:56Z
  Ended:       2026-09-16T15:44:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7769984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss65f27i0n6ratrctdqlr1g1ugtvfe8-v8e3q9j6q4c5y2d", pod="router-6c9c647b4-m7ffl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss65f27i0n6ratrctdqlr1g1ugtvfe8-v8e3q9j6q4c5y2d/router-6c9c647b4-m7ffl has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-16T15:26:56Z
  Ended:       2026-09-16T15:48:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7769984-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss65f7fgkikv4rdlo37t3t3p2g7ndo6-e7h0d1m9e7x8u6b", pod="router-776ccdc86d-stwfg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss65f7fgkikv4rdlo37t3t3p2g7ndo6-e7h0d1m9e7x8u6b/router-776ccdc86d-stwfg has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-16T15:26:56Z
  Ended:       2026-09-16T15:54:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss65fb8rtujqr3jelbpv31ep2qkd2cj-m1n3i0x2v3m7r0r", pod="router-58d4f4f4c-spsms", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss65fb8rtujqr3jelbpv31ep2qkd2cj-m1n3i0x2v3m7r0r/router-58d4f4f4c-spsms has been in a non-ready state for longer than 5 minutes.

time=2026-09-16T14:44:36.842Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T14:44:37.777Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-16T14:44:38.583Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=00e0f6eaf1a87ba23e25957419627b2f2bf902c90c40b4a9a0680ec1b94026a9 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1035008-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F00e0f6eaf1a87ba23e25957419627b2f2bf902c90c40b4a9a0680ec1b94026a9
time=2026-09-16T15:14:36.624Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000eec980>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-s5qsjcnlw5bb" to finish creating: GET https://rp.j3991936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91f7755d-f9d4-4444-beb5-36084a84bad6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91f7755d-f9d4-4444-beb5-36084a84bad6",
      "name": "91f7755d-f9d4-4444-beb5-36084a84bad6",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:08.748392201Z",
      "endTime": "2026-09-16T15:36:08.491792719Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000eec980>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-s5qsjcnlw5bb" to finish creating: GET https://rp.j3991936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91f7755d-f9d4-4444-beb5-36084a84bad6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/91f7755d-f9d4-4444-beb5-36084a84bad6",
      "name": "91f7755d-f9d4-4444-beb5-36084a84bad6",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:08.748392201Z",
      "endTime": "2026-09-16T15:36:08.491792719Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:09:29Z
  Ended:       2026-09-16T15:39:29Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3991936-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3991936-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e74c20>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xzp2sp8wgrn2" to finish creating: GET https://rp.j2103808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c63d988d-abe6-4ea1-9d3f-acc959502c90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c63d988d-abe6-4ea1-9d3f-acc959502c90",
      "name": "c63d988d-abe6-4ea1-9d3f-acc959502c90",
      "status": "Failed",
      "startTime": "2026-09-16T15:04:26.332691883Z",
      "endTime": "2026-09-16T15:33:30.087028134Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e74c20>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xzp2sp8wgrn2" to finish creating: GET https://rp.j2103808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c63d988d-abe6-4ea1-9d3f-acc959502c90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c63d988d-abe6-4ea1-9d3f-acc959502c90",
      "name": "c63d988d-abe6-4ea1-9d3f-acc959502c90",
      "status": "Failed",
      "startTime": "2026-09-16T15:04:26.332691883Z",
      "endTime": "2026-09-16T15:33:30.087028134Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001712108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-z7jd9d5bkv4x" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe3c0be1-f3f8-476b-b13f-1ceef9fd8976
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe3c0be1-f3f8-476b-b13f-1ceef9fd8976",
      "name": "fe3c0be1-f3f8-476b-b13f-1ceef9fd8976",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:43.887981496Z",
      "endTime": "2026-09-16T15:47:44.543137714Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss654fr7qth67si53recd86bd0moi28 still exists (deletion dispatched at 2026-09-16T15:23:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001712108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-z7jd9d5bkv4x" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe3c0be1-f3f8-476b-b13f-1ceef9fd8976
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe3c0be1-f3f8-476b-b13f-1ceef9fd8976",
      "name": "fe3c0be1-f3f8-476b-b13f-1ceef9fd8976",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:43.887981496Z",
      "endTime": "2026-09-16T15:47:44.543137714Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss654fr7qth67si53recd86bd0moi28 still exists (deletion dispatched at 2026-09-16T15:23:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000fcac20>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-jf5j95vczmzx" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c06c517-46e6-4d48-93f9-026c39ae1338
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c06c517-46e6-4d48-93f9-026c39ae1338",
      "name": "9c06c517-46e6-4d48-93f9-026c39ae1338",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:48.234554059Z",
      "endTime": "2026-09-16T15:34:54.47386127Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000fcac20>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-jf5j95vczmzx" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c06c517-46e6-4d48-93f9-026c39ae1338
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c06c517-46e6-4d48-93f9-026c39ae1338",
      "name": "9c06c517-46e6-4d48-93f9-026c39ae1338",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:48.234554059Z",
      "endTime": "2026-09-16T15:34:54.47386127Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e3a9a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9r6rpqjgz268" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb680db4-320e-4707-9f4b-0f2cd699ec63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb680db4-320e-4707-9f4b-0f2cd699ec63",
      "name": "cb680db4-320e-4707-9f4b-0f2cd699ec63",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:59.161824665Z",
      "endTime": "2026-09-16T15:35:04.514571365Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000e3a9a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9r6rpqjgz268" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb680db4-320e-4707-9f4b-0f2cd699ec63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb680db4-320e-4707-9f4b-0f2cd699ec63",
      "name": "cb680db4-320e-4707-9f4b-0f2cd699ec63",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:59.161824665Z",
      "endTime": "2026-09-16T15:35:04.514571365Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 24m8s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001094a00>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-nnpsp9gzr6sk" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3a50c81-c043-4ac7-a378-fa722f63c462
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3a50c81-c043-4ac7-a378-fa722f63c462",
      "name": "d3a50c81-c043-4ac7-a378-fa722f63c462",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:49.08539059Z",
      "endTime": "2026-09-16T15:34:54.502329591Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m17s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001094a00>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-nnpsp9gzr6sk" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3a50c81-c043-4ac7-a378-fa722f63c462
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d3a50c81-c043-4ac7-a378-fa722f63c462",
      "name": "d3a50c81-c043-4ac7-a378-fa722f63c462",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:49.08539059Z",
      "endTime": "2026-09-16T15:34:54.502329591Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m17s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001532060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-g8h2hwb78d6v" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acb2e182-ba85-4e97-b275-c6baf85458ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acb2e182-ba85-4e97-b275-c6baf85458ad",
      "name": "acb2e182-ba85-4e97-b275-c6baf85458ad",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:20.960856273Z",
      "endTime": "2026-09-16T15:46:24.513434488Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64nns1ma3p3sappn3d83v1kq4u4mc still exists (deletion dispatched at 2026-09-16T15:22:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001532060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-g8h2hwb78d6v" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acb2e182-ba85-4e97-b275-c6baf85458ad
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/acb2e182-ba85-4e97-b275-c6baf85458ad",
      "name": "acb2e182-ba85-4e97-b275-c6baf85458ad",
      "status": "Failed",
      "startTime": "2026-09-16T15:22:20.960856273Z",
      "endTime": "2026-09-16T15:46:24.513434488Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64nns1ma3p3sappn3d83v1kq4u4mc still exists (deletion dispatched at 2026-09-16T15:22:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv"
Unexpected error:
    <*fmt.wrapError | 0xc000d846a0>: 
    failed to create NodePool np-96ddmv: failed waiting for nodepool="np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv" in resourcegroup="rg-candidate-5-0-dxgjw5725ck7" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "name": "75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:26.413052673Z",
      "endTime": "2026-09-16T15:24:04.373688035Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv"
Unexpected error:
    <*fmt.wrapError | 0xc000d846a0>: 
    failed to create NodePool np-96ddmv: failed waiting for nodepool="np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv" in resourcegroup="rg-candidate-5-0-dxgjw5725ck7" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "name": "75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:26.413052673Z",
      "endTime": "2026-09-16T15:24:04.373688035Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001880090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-pp5nvw5grtc7" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/286d07fb-7bc7-4ce1-aafb-d75cdb5dbbfc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/286d07fb-7bc7-4ce1-aafb-d75cdb5dbbfc",
      "name": "286d07fb-7bc7-4ce1-aafb-d75cdb5dbbfc",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:13.971647999Z",
      "endTime": "2026-09-16T15:47:14.615898434Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss65qamt1ukuh6foc29hal8kisg465u still exists (deletion dispatched at 2026-09-16T15:23:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001880090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-pp5nvw5grtc7" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/286d07fb-7bc7-4ce1-aafb-d75cdb5dbbfc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/286d07fb-7bc7-4ce1-aafb-d75cdb5dbbfc",
      "name": "286d07fb-7bc7-4ce1-aafb-d75cdb5dbbfc",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:13.971647999Z",
      "endTime": "2026-09-16T15:47:14.615898434Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss65qamt1ukuh6foc29hal8kisg465u still exists (deletion dispatched at 2026-09-16T15:23:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a101b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-d7xf2jghkngg" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe4bbd3c-2ce0-4443-b660-f1afc9b9ff48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe4bbd3c-2ce0-4443-b660-f1afc9b9ff48",
      "name": "fe4bbd3c-2ce0-4443-b660-f1afc9b9ff48",
      "status": "Failed",
      "startTime": "2026-09-16T15:25:39.570793353Z",
      "endTime": "2026-09-16T15:49:44.508102649Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64gl4enpi00gf4b4hmbe2u4tjukt5 still exists (deletion dispatched at 2026-09-16T15:25:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a101b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-d7xf2jghkngg" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe4bbd3c-2ce0-4443-b660-f1afc9b9ff48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fe4bbd3c-2ce0-4443-b660-f1afc9b9ff48",
      "name": "fe4bbd3c-2ce0-4443-b660-f1afc9b9ff48",
      "status": "Failed",
      "startTime": "2026-09-16T15:25:39.570793353Z",
      "endTime": "2026-09-16T15:49:44.508102649Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64gl4enpi00gf4b4hmbe2u4tjukt5 still exists (deletion dispatched at 2026-09-16T15:25:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001660078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-swrr5k" in resourcegroup="rg-zstream-upgrade-4-21-xx2vdc7swlps" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81df39bb-b3b6-42b1-a83c-57dc3461bd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81df39bb-b3b6-42b1-a83c-57dc3461bd95",
      "name": "81df39bb-b3b6-42b1-a83c-57dc3461bd95",
      "status": "Failed",
      "startTime": "2026-09-16T15:30:38.197952822Z",
      "endTime": "2026-09-16T15:54:44.460453861Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64iqhe77hj84b81d1vg9nt6d0dhsr still exists (deletion dispatched at 2026-09-16T15:30:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001660078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-swrr5k" in resourcegroup="rg-zstream-upgrade-4-21-xx2vdc7swlps" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81df39bb-b3b6-42b1-a83c-57dc3461bd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/81df39bb-b3b6-42b1-a83c-57dc3461bd95",
      "name": "81df39bb-b3b6-42b1-a83c-57dc3461bd95",
      "status": "Failed",
      "startTime": "2026-09-16T15:30:38.197952822Z",
      "endTime": "2026-09-16T15:54:44.460453861Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64iqhe77hj84b81d1vg9nt6d0dhsr still exists (deletion dispatched at 2026-09-16T15:30:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012e40>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-wbh9jqvv44p7" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9db0308e-c5ac-4693-b367-d76dc77af863
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9db0308e-c5ac-4693-b367-d76dc77af863",
      "name": "9db0308e-c5ac-4693-b367-d76dc77af863",
      "status": "Failed",
      "startTime": "2026-09-16T15:29:44.443897591Z",
      "endTime": "2026-09-16T15:53:44.471117038Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64i7i766p1sol3r0dalpub5p7l53h still exists (deletion dispatched at 2026-09-16T15:29:46Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012e40>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-wbh9jqvv44p7" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9db0308e-c5ac-4693-b367-d76dc77af863
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9db0308e-c5ac-4693-b367-d76dc77af863",
      "name": "9db0308e-c5ac-4693-b367-d76dc77af863",
      "status": "Failed",
      "startTime": "2026-09-16T15:29:44.443897591Z",
      "endTime": "2026-09-16T15:53:44.471117038Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss64i7i766p1sol3r0dalpub5p7l53h still exists (deletion dispatched at 2026-09-16T15:29:46Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001006f60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-gpg49rkjvb2m" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c60438e7-1f3d-4f9d-a2a0-7dd97f6e3d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c60438e7-1f3d-4f9d-a2a0-7dd97f6e3d9b",
      "name": "c60438e7-1f3d-4f9d-a2a0-7dd97f6e3d9b",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:45.793133162Z",
      "endTime": "2026-09-16T15:34:54.478500072Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001006f60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-gpg49rkjvb2m" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c60438e7-1f3d-4f9d-a2a0-7dd97f6e3d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c60438e7-1f3d-4f9d-a2a0-7dd97f6e3d9b",
      "name": "c60438e7-1f3d-4f9d-a2a0-7dd97f6e3d9b",
      "status": "Failed",
      "startTime": "2026-09-16T15:05:45.793133162Z",
      "endTime": "2026-09-16T15:34:54.478500072Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004e6380>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-z42dcrsxlj77" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6cbd1f6e-1e54-401b-8dd5-b1aea43e3a87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6cbd1f6e-1e54-401b-8dd5-b1aea43e3a87",
      "name": "6cbd1f6e-1e54-401b-8dd5-b1aea43e3a87",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:22.469453633Z",
      "endTime": "2026-09-16T15:36:24.366773591Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0004e6380>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-z42dcrsxlj77" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6cbd1f6e-1e54-401b-8dd5-b1aea43e3a87
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6cbd1f6e-1e54-401b-8dd5-b1aea43e3a87",
      "name": "6cbd1f6e-1e54-401b-8dd5-b1aea43e3a87",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:22.469453633Z",
      "endTime": "2026-09-16T15:36:24.366773591Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011a4060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-k6j48pxdpt2p" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6e3e29a-385a-455a-b41d-893521143636
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6e3e29a-385a-455a-b41d-893521143636",
      "name": "a6e3e29a-385a-455a-b41d-893521143636",
      "status": "Failed",
      "startTime": "2026-09-16T15:28:48.608443563Z",
      "endTime": "2026-09-16T15:52:54.427688455Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss657krpigsh0ful51ank8n8kbiubtm still exists (deletion dispatched at 2026-09-16T15:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011a4060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-k6j48pxdpt2p" to finish deleting: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6e3e29a-385a-455a-b41d-893521143636
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6e3e29a-385a-455a-b41d-893521143636",
      "name": "a6e3e29a-385a-455a-b41d-893521143636",
      "status": "Failed",
      "startTime": "2026-09-16T15:28:48.608443563Z",
      "endTime": "2026-09-16T15:52:54.427688455Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss657krpigsh0ful51ank8n8kbiubtm still exists (deletion dispatched at 2026-09-16T15:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010207a0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-z5wdv6h4vdnz" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba551b96-dd27-4bf9-b784-21cb42b2d916
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba551b96-dd27-4bf9-b784-21cb42b2d916",
      "name": "ba551b96-dd27-4bf9-b784-21cb42b2d916",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:31.358252502Z",
      "endTime": "2026-09-16T15:37:34.476730603Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010207a0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-z5wdv6h4vdnz" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba551b96-dd27-4bf9-b784-21cb42b2d916
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ba551b96-dd27-4bf9-b784-21cb42b2d916",
      "name": "ba551b96-dd27-4bf9-b784-21cb42b2d916",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:31.358252502Z",
      "endTime": "2026-09-16T15:37:34.476730603Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:28:20Z
  Ended:       2026-09-16T15:40:21Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j8680064-svc", component="backend", name="dataplaneoidcfederation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue dataplaneoidcfederation has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:24:31Z
  Ended:       2026-09-16T15:42:32Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8680064-svc", component="backend", name="dataplaneoidcfederation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue dataplaneoidcfederation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:16:31Z
  Ended:       2026-09-16T16:05:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8680064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss64a2ngrrh8v43g4224kkvsoq6vf20-l5z4r5n6l2r7j5q", pod="router-686dcf6999-wvnqc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss64a2ngrrh8v43g4224kkvsoq6vf20-l5z4r5n6l2r7j5q/router-686dcf6999-wvnqc has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T15:17:31Z
  Ended:       2026-09-16T15:49:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8680064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss64lsro1oea3d8uqc5rdlv4i8bfn9s-n0h3v6f4v6w5e2y", pod="router-656766fdcc-4gn7f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss64lsro1oea3d8uqc5rdlv4i8bfn9s-n0h3v6f4v6w5e2y/router-656766fdcc-4gn7f has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T15:18:31Z
  Ended:       2026-09-16T15:45:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8680064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss64j8k24ndk4j4jets482r57jmq464-private-ingress", pod="router-8698f9c9d5-f9464", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss64j8k24ndk4j4jets482r57jmq464-private-ingress/router-8698f9c9d5-f9464 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T15:18:31Z
  Ended:       2026-09-16T15:44:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8680064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss64jdehrrlcdafsmr824r6gu6d4v3i-k5y9i2m2i7c3a6z", pod="router-684b9dbc6c-mknm9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss64jdehrrlcdafsmr824r6gu6d4v3i-k5y9i2m2i7c3a6z/router-684b9dbc6c-mknm9 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0017c8048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-jxshqr" in resourcegroup="rg-zstream-upgrade-4-20-h67wq8dhmgts" to finish deleting: GET https://rp.j8039808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b1e26b2-b451-4f5e-9c4e-f445f49d6b86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b1e26b2-b451-4f5e-9c4e-f445f49d6b86",
      "name": "5b1e26b2-b451-4f5e-9c4e-f445f49d6b86",
      "status": "Failed",
      "startTime": "2026-09-16T15:13:10.410853288Z",
      "endTime": "2026-09-16T15:37:12.377929551Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5vh867atil535pho6oarttl9i35ou still exists (deletion dispatched at 2026-09-16T15:13:20Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0017c8048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-jxshqr" in resourcegroup="rg-zstream-upgrade-4-20-h67wq8dhmgts" to finish deleting: GET https://rp.j8039808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b1e26b2-b451-4f5e-9c4e-f445f49d6b86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5b1e26b2-b451-4f5e-9c4e-f445f49d6b86",
      "name": "5b1e26b2-b451-4f5e-9c4e-f445f49d6b86",
      "status": "Failed",
      "startTime": "2026-09-16T15:13:10.410853288Z",
      "endTime": "2026-09-16T15:37:12.377929551Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5vh867atil535pho6oarttl9i35ou still exists (deletion dispatched at 2026-09-16T15:13:20Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:144]: node pool "np-1" should have exactly one active version after initial creation
Expected
    <[]*armredhatopenshifthcp.NodePoolActiveVersion | len:0, cap:0>: ...
to have length 1
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:144]: node pool "np-1" should have exactly one active version after initial creation
Expected
    <[]*armredhatopenshifthcp.NodePoolActiveVersion | len:0, cap:0>: ...
to have length 1

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019b8288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-r7ljx995hlwf" to finish deleting: GET https://rp.j8039808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/45f270e6-1026-464f-92a5-1c2c1a33f197
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/45f270e6-1026-464f-92a5-1c2c1a33f197",
      "name": "45f270e6-1026-464f-92a5-1c2c1a33f197",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:09.364011371Z",
      "endTime": "2026-09-16T15:36:12.497199905Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5vol9udvmi9p399pnj4tdfvicpu9k still exists (deletion dispatched at 2026-09-16T15:12:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019b8288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-r7ljx995hlwf" to finish deleting: GET https://rp.j8039808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/45f270e6-1026-464f-92a5-1c2c1a33f197
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/45f270e6-1026-464f-92a5-1c2c1a33f197",
      "name": "45f270e6-1026-464f-92a5-1c2c1a33f197",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:09.364011371Z",
      "endTime": "2026-09-16T15:36:12.497199905Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5vol9udvmi9p399pnj4tdfvicpu9k still exists (deletion dispatched at 2026-09-16T15:12:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00133c168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-8q54sb" in resourcegroup="rg-candidate-4-21-96n8w9b9nmcr" to finish deleting: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8145a55-f885-4824-ade4-c78ded06b41b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8145a55-f885-4824-ade4-c78ded06b41b",
      "name": "d8145a55-f885-4824-ade4-c78ded06b41b",
      "status": "Failed",
      "startTime": "2026-09-16T15:09:44.33584903Z",
      "endTime": "2026-09-16T15:33:51.215190325Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5qm47uq9tl4u0n0hlgrdb7qttgv6e still exists (deletion dispatched at 2026-09-16T15:09:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00133c168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-8q54sb" in resourcegroup="rg-candidate-4-21-96n8w9b9nmcr" to finish deleting: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8145a55-f885-4824-ade4-c78ded06b41b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d8145a55-f885-4824-ade4-c78ded06b41b",
      "name": "d8145a55-f885-4824-ade4-c78ded06b41b",
      "status": "Failed",
      "startTime": "2026-09-16T15:09:44.33584903Z",
      "endTime": "2026-09-16T15:33:51.215190325Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5qm47uq9tl4u0n0hlgrdb7qttgv6e still exists (deletion dispatched at 2026-09-16T15:09:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001654180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-vwpvr2srntsx" to finish deleting: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9690275d-f0a2-4269-97f3-ff42287a50a3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9690275d-f0a2-4269-97f3-ff42287a50a3",
      "name": "9690275d-f0a2-4269-97f3-ff42287a50a3",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:51.225823952Z",
      "endTime": "2026-09-16T15:32:51.218856846Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5qjn2q2n7cj829v2pum0bhr8rsof2 still exists (deletion dispatched at 2026-09-16T15:08:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001654180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-vwpvr2srntsx" to finish deleting: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9690275d-f0a2-4269-97f3-ff42287a50a3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9690275d-f0a2-4269-97f3-ff42287a50a3",
      "name": "9690275d-f0a2-4269-97f3-ff42287a50a3",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:51.225823952Z",
      "endTime": "2026-09-16T15:32:51.218856846Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss5qjn2q2n7cj829v2pum0bhr8rsof2 still exists (deletion dispatched at 2026-09-16T15:08:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:403]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc000dfe180>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-kp7j9psbb8sj" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/284aa271-80bb-4df5-aaf0-7131d6eceb09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/284aa271-80bb-4df5-aaf0-7131d6eceb09",
      "name": "284aa271-80bb-4df5-aaf0-7131d6eceb09",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:21.082535677Z",
      "endTime": "2026-09-16T15:13:01.614774274Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:403]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc000dfe180>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-kp7j9psbb8sj" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/284aa271-80bb-4df5-aaf0-7131d6eceb09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/284aa271-80bb-4df5-aaf0-7131d6eceb09",
      "name": "284aa271-80bb-4df5-aaf0-7131d6eceb09",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:21.082535677Z",
      "endTime": "2026-09-16T15:13:01.614774274Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000fec640>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9qq8dbbbpq99" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c197dc3a-1727-4935-ab5c-4e20220390ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c197dc3a-1727-4935-ab5c-4e20220390ea",
      "name": "c197dc3a-1727-4935-ab5c-4e20220390ea",
      "status": "Failed",
      "startTime": "2026-09-16T14:44:47.197696304Z",
      "endTime": "2026-09-16T15:13:51.528371822Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 17m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000fec640>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9qq8dbbbpq99" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c197dc3a-1727-4935-ab5c-4e20220390ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c197dc3a-1727-4935-ab5c-4e20220390ea",
      "name": "c197dc3a-1727-4935-ab5c-4e20220390ea",
      "status": "Failed",
      "startTime": "2026-09-16T14:44:47.197696304Z",
      "endTime": "2026-09-16T15:13:51.528371822Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 17m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0011eef40>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gn4q6q4pqf7b" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406",
      "name": "c302cb08-5816-4f3c-a786-daa45a7d6406",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:22.418838081Z",
      "endTime": "2026-09-16T15:13:01.450563178Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0011eef40>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gn4q6q4pqf7b" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406",
      "name": "c302cb08-5816-4f3c-a786-daa45a7d6406",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:22.418838081Z",
      "endTime": "2026-09-16T15:13:01.450563178Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 20 time(s) (2 unknown, 18 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:02:01Z
  Ended:       2026-09-16T15:18:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4331648-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss5qrj6l3t7svqqgghv0p32ifilmtgo-private-kas", pod="router-74979f6ddd-l297b", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss5qrj6l3t7svqqgghv0p32ifilmtgo-private-kas/router-74979f6ddd-l297b has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T15:02:01Z
  Ended:       2026-09-16T15:18:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4331648-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss5qnvpmpmfksiqumamsk6f4a5n8h97-i8m9u7o2i9m1s4l", pod="router-d798bf795-r5qm8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss5qnvpmpmfksiqumamsk6f4a5n8h97-i8m9u7o2i9m1s4l/router-d798bf795-r5qm8 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7e2c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-hk9kwb8xmkfd" to finish creating: GET https://rp.j3700352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f5beb08-52f4-4ca9-8de7-6fbc3e2fae17
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f5beb08-52f4-4ca9-8de7-6fbc3e2fae17",
      "name": "1f5beb08-52f4-4ca9-8de7-6fbc3e2fae17",
      "status": "Failed",
      "startTime": "2026-09-16T13:59:17.257119371Z",
      "endTime": "2026-09-16T14:28:24.535752776Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7e2c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-hk9kwb8xmkfd" to finish creating: GET https://rp.j3700352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f5beb08-52f4-4ca9-8de7-6fbc3e2fae17
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f5beb08-52f4-4ca9-8de7-6fbc3e2fae17",
      "name": "1f5beb08-52f4-4ca9-8de7-6fbc3e2fae17",
      "status": "Failed",
      "startTime": "2026-09-16T13:59:17.257119371Z",
      "endTime": "2026-09-16T14:28:24.535752776Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000acc200>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-hld8zl98shzl" to finish creating: GET https://rp.j3700352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56519874-edc6-422e-930e-b7dc2d33fa54
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56519874-edc6-422e-930e-b7dc2d33fa54",
      "name": "56519874-edc6-422e-930e-b7dc2d33fa54",
      "status": "Failed",
      "startTime": "2026-09-16T13:59:08.637456605Z",
      "endTime": "2026-09-16T14:28:14.536216515Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m26s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000acc200>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-hld8zl98shzl" to finish creating: GET https://rp.j3700352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56519874-edc6-422e-930e-b7dc2d33fa54
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/56519874-edc6-422e-930e-b7dc2d33fa54",
      "name": "56519874-edc6-422e-930e-b7dc2d33fa54",
      "status": "Failed",
      "startTime": "2026-09-16T13:59:08.637456605Z",
      "endTime": "2026-09-16T14:28:14.536216515Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m26s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T14:14:40Z
  Ended:       2026-09-16T14:41:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3700352-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss55dasvijc9n7rk5tev7k98q8t38ut-f3o4a6n6m5t7w8i", pod="router-6594467575-v5mmd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss55dasvijc9n7rk5tev7k98q8t38ut-f3o4a6n6m5t7w8i/router-6594467575-v5mmd has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T14:14:40Z
  Ended:       2026-09-16T14:41:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3700352-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss55beaf4jnkm30gd64lt3c9jhq19e4-private-ingress", pod="router-7db4c964-ls58f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss55beaf4jnkm30gd64lt3c9jhq19e4-private-ingress/router-7db4c964-ls58f has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0008cd0a0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jvwfr7" in resourcegroup="rg-np-skip-minor-jvwfr7-7qvvgr9wwfbq" to finish creating: GET https://rp.j2424320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "name": "6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "status": "Failed",
      "startTime": "2026-09-16T14:13:13.201859753Z",
      "endTime": "2026-09-16T14:32:13.373200335Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0008cd0a0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jvwfr7" in resourcegroup="rg-np-skip-minor-jvwfr7-7qvvgr9wwfbq" to finish creating: GET https://rp.j2424320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "name": "6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "status": "Failed",
      "startTime": "2026-09-16T14:13:13.201859753Z",
      "endTime": "2026-09-16T14:32:13.373200335Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00098aa40>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-6th5jd8nbg5s" to finish creating: GET https://rp.j2424320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd1a9666-9126-4a24-9b9a-2799ae9c069b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd1a9666-9126-4a24-9b9a-2799ae9c069b",
      "name": "bd1a9666-9126-4a24-9b9a-2799ae9c069b",
      "status": "Failed",
      "startTime": "2026-09-16T13:57:52.148119457Z",
      "endTime": "2026-09-16T14:26:53.36802381Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, community-operators-catalog, hosted-cluster-config-operator, catalog-operator, redhat-operators-catalog, ingress-operator, cluster-image-registry-operator, openshift-route-controller-manager, cluster-node-tuning-operator, dns-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, packageserver, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00098aa40>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-6th5jd8nbg5s" to finish creating: GET https://rp.j2424320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd1a9666-9126-4a24-9b9a-2799ae9c069b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd1a9666-9126-4a24-9b9a-2799ae9c069b",
      "name": "bd1a9666-9126-4a24-9b9a-2799ae9c069b",
      "status": "Failed",
      "startTime": "2026-09-16T13:57:52.148119457Z",
      "endTime": "2026-09-16T14:26:53.36802381Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, community-operators-catalog, hosted-cluster-config-operator, catalog-operator, redhat-operators-catalog, ingress-operator, cluster-image-registry-operator, openshift-route-controller-manager, cluster-node-tuning-operator, dns-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, packageserver, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-16T13:31:49.803Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-09-16T13:31:51.360Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-09-16T13:31:52.622Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=6a654e03e0d8cd3ba2dcf4b7ca7ea7b9161da4e2734cccce092b3f0e303e2fe9 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6373376-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6a654e03e0d8cd3ba2dcf4b7ca7ea7b9161da4e2734cccce092b3f0e303e2fe9
time=2026-09-16T13:40:24.258Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6373376-svc/providers/Microsoft.Resources/deployments/6a654e03e0d8cd3ba2dcf4b7ca7ea7b9161da4e2734cccce092b3f0e303e2fe9/operationStatuses/08584120413735528505\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6373376/providers/Microsoft.Resources/deployments/rp_cosmos_db\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6373376/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j6373376/sqlRoleAssignments/3a943d25-7e89-52ef-bdd2-d3814561859c\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"BadRequest\\\",\\r\\n                \\\"message\\\": \\\"The collection with name [Resources] in provided scope [/dbs/arohcpci01-rp-j6373376/colls/Resources] could not be found.\\\\r\\\\nActivityId: 385582e2-ce0e-4904-a9f2-7469a3089fa3, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jz5x7r with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000cb6700>: 
    failed to create HCP cluster np-version-upgrade-cluster-jz5x7r: failed waiting for cluster="np-version-upgrade-cluster-jz5x7r" in resourcegroup="rg-np-version-upgrade-jz5x7r-xds97j5kvxhn" to finish creating: GET https://rp.j9619840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b59189b4-8e72-4fa2-92fe-a2a6f5dd2eb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b59189b4-8e72-4fa2-92fe-a2a6f5dd2eb3",
      "name": "b59189b4-8e72-4fa2-92fe-a2a6f5dd2eb3",
      "status": "Failed",
      "startTime": "2026-09-16T13:41:36.451206458Z",
      "endTime": "2026-09-16T14:10:39.504401971Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jz5x7r with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000cb6700>: 
    failed to create HCP cluster np-version-upgrade-cluster-jz5x7r: failed waiting for cluster="np-version-upgrade-cluster-jz5x7r" in resourcegroup="rg-np-version-upgrade-jz5x7r-xds97j5kvxhn" to finish creating: GET https://rp.j9619840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b59189b4-8e72-4fa2-92fe-a2a6f5dd2eb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b59189b4-8e72-4fa2-92fe-a2a6f5dd2eb3",
      "name": "b59189b4-8e72-4fa2-92fe-a2a6f5dd2eb3",
      "status": "Failed",
      "startTime": "2026-09-16T13:41:36.451206458Z",
      "endTime": "2026-09-16T14:10:39.504401971Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T14:03:47Z
  Ended:       2026-09-16T14:33:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9619840-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9619840/providers/microsoft.monitor/accounts/hcps-j9619840", namespace="ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aaf39014-4158-4621-b188-69047e816d57"
  Description: Pod ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T14:02:53Z
  Ended:       2026-09-16T14:27:49Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9619840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9619840/providers/microsoft.monitor/accounts/hcps-j9619840", namespace="ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r", pod="arohcp-monitor-kube-state-metrics-574f657d96-q22d9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T14:02:53Z
  Ended:       2026-09-16T14:22:48Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9619840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9619840/providers/microsoft.monitor/accounts/hcps-j9619840", namespace="ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r", pod="arohcp-monitor-kube-state-metrics-574f657d96-q22d9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T14:16:51Z
  Ended:       2026-09-16T14:34:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9619840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9619840/providers/microsoft.monitor/accounts/hcps-j9619840", namespace="ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r", pod="arohcp-monitor-kube-state-metrics-574f657d96-q22d9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 8 time(s) (1 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T13:52:47Z
  Ended:       2026-09-16T14:11:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9619840-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss4t6obva3desivmr8hmkd7a8qlvbhm-n2t8z7n2e0x3c5e", pod="router-d9fc9d667-sk5dl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss4t6obva3desivmr8hmkd7a8qlvbhm-n2t8z7n2e0x3c5e/router-d9fc9d667-sk5dl has been in a non-ready state for longer than 5 minutes.

time=2026-09-16T12:42:02.218Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T12:42:02.867Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T12:42:03.803Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=f62142d1821bad4afc9418f7e325718ebc679cc0887c30afb66e545abef06520 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1791104-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff62142d1821bad4afc9418f7e325718ebc679cc0887c30afb66e545abef06520
time=2026-09-16T13:12:01.951Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0013c07a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-42qs67sfvs6r" to finish creating: GET https://rp.j9096832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62ed0453-0f1a-4cb5-b502-4690d5fe8e28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62ed0453-0f1a-4cb5-b502-4690d5fe8e28",
      "name": "62ed0453-0f1a-4cb5-b502-4690d5fe8e28",
      "status": "Failed",
      "startTime": "2026-09-16T12:56:55.481514146Z",
      "endTime": "2026-09-16T13:25:58.518527402Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0013c07a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-42qs67sfvs6r" to finish creating: GET https://rp.j9096832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62ed0453-0f1a-4cb5-b502-4690d5fe8e28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62ed0453-0f1a-4cb5-b502-4690d5fe8e28",
      "name": "62ed0453-0f1a-4cb5-b502-4690d5fe8e28",
      "status": "Failed",
      "startTime": "2026-09-16T12:56:55.481514146Z",
      "endTime": "2026-09-16T13:25:58.518527402Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-d52gscxjdr6c/cluster-candidate-4-21-wc8v7p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f7c4c0>: 
    failed to create HCP cluster cluster-candidate-4-21-wc8v7p: failed waiting for cluster="cluster-candidate-4-21-wc8v7p" in resourcegroup="rg-candidate-4-21-d52gscxjdr6c" to finish creating: GET https://rp.j9096832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecd643be-f810-4322-9880-6c4747c1f655
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecd643be-f810-4322-9880-6c4747c1f655",
      "name": "ecd643be-f810-4322-9880-6c4747c1f655",
      "status": "Failed",
      "startTime": "2026-09-16T12:56:57.32608726Z",
      "endTime": "2026-09-16T13:25:58.487825893Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m4s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-d52gscxjdr6c/cluster-candidate-4-21-wc8v7p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f7c4c0>: 
    failed to create HCP cluster cluster-candidate-4-21-wc8v7p: failed waiting for cluster="cluster-candidate-4-21-wc8v7p" in resourcegroup="rg-candidate-4-21-d52gscxjdr6c" to finish creating: GET https://rp.j9096832.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecd643be-f810-4322-9880-6c4747c1f655
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ecd643be-f810-4322-9880-6c4747c1f655",
      "name": "ecd643be-f810-4322-9880-6c4747c1f655",
      "status": "Failed",
      "startTime": "2026-09-16T12:56:57.32608726Z",
      "endTime": "2026-09-16T13:25:58.487825893Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m4s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-16T12:31:03.652Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T12:31:04.249Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T12:31:05.229Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=c3798ec18923b6e7d63ee7828445bbe6248d8a37c22e598b697019726ef5af5a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3099520-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc3798ec18923b6e7d63ee7828445bbe6248d8a37c22e598b697019726ef5af5a
time=2026-09-16T13:01:03.275Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012e60a0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012e60a0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc00072e820>: 
    failed to get workspace hcps-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/hcps-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc00072e820>: 
    failed to get workspace hcps-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/hcps-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-j3128064 in resource group hcp-underlay-ci01-j3128064
Unexpected error:
    <*fmt.wrapError | 0xc0010489e0>: 
    failed to get workspace services-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/services-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-j3128064 in resource group hcp-underlay-ci01-j3128064
Unexpected error:
    <*fmt.wrapError | 0xc0010489e0>: 
    failed to get workspace services-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/services-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc0017cdba0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:34702->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc0017cdba0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:34702->172.30.0.10:53: read: connection refused
    ...

alert fired 44 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T13:01:29Z
  Ended:       2026-09-16T13:27:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss457poa72ja0btdj1ld3p1ldvs4gtt-b7a5e9g4f3j4n2u", pod="kube-state-metrics-hcp-8654bffc85-smj2c", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss457poa72ja0btdj1ld3p1ldvs4gtt-b7a5e9g4f3j4n2u/kube-state-metrics-hcp-8654bffc85-smj2c has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:33:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45nv36i1dc0a9pkmr8als6f6o3fig-l7l2b5f0n2d0s0z", pod="kube-state-metrics-hcp-8654bffc85-s6k2r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45nv36i1dc0a9pkmr8als6f6o3fig-l7l2b5f0n2d0s0z/kube-state-metrics-hcp-8654bffc85-s6k2r has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:32:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45a9ct6a0mh48hpkkmm7h44ov42vt-m4m6l3b4q9e9f9l", pod="kube-state-metrics-hcp-8654bffc85-6czgh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45a9ct6a0mh48hpkkmm7h44ov42vt-m4m6l3b4q9e9f9l/kube-state-metrics-hcp-8654bffc85-6czgh has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:20:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45daaanj7b9tf929bmahf1ptl7bk8-oidc-wi-cluster", pod="kube-state-metrics-hcp-8654bffc85-kglmw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45daaanj7b9tf929bmahf1ptl7bk8-oidc-wi-cluster/kube-state-metrics-hcp-8654bffc85-kglmw has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:17:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45e9nd54k58re0kpbnm8ce4j9mk0j-sre-hcp-cluster", pod="kube-state-metrics-hcp-8654bffc85-d5xl8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45e9nd54k58re0kpbnm8ce4j9mk0j-sre-hcp-cluster/kube-state-metrics-hcp-8654bffc85-d5xl8 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:20:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss455n64tgfov79qogjllrl433lkcdu-a4n5t3g6x1e8t8w", pod="kube-state-metrics-hcp-8654bffc85-l76th", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss455n64tgfov79qogjllrl433lkcdu-a4n5t3g6x1e8t8w/kube-state-metrics-hcp-8654bffc85-l76th has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:11:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss457qlr01ukp4lj3sk5hot62sn7k8r-u3b8i6t7f6q2z8w", pod="kube-state-metrics-hcp-8654bffc85-prmvm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss457qlr01ukp4lj3sk5hot62sn7k8r-u3b8i6t7f6q2z8w/kube-state-metrics-hcp-8654bffc85-prmvm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45aebd49uls1uivkklt2gp3aq0a75-p4k0j6h6l2a3p5k", pod="kube-state-metrics-hcp-8654bffc85-ptg8g", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45aebd49uls1uivkklt2gp3aq0a75-p4k0j6h6l2a3p5k/kube-state-metrics-hcp-8654bffc85-ptg8g has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:30:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45d6n0244mp2oimtij4qn88f83qjc-l2o2i1p1b9i7k1b", pod="kube-state-metrics-hcp-8654bffc85-vs7hk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45d6n0244mp2oimtij4qn88f83qjc-l2o2i1p1b9i7k1b/kube-state-metrics-hcp-8654bffc85-vs7hk has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-16T13:02:31Z
  Ended:       2026-09-16T13:21:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss453dqari5t3de9k8dotol85jfinfd-q0j6a1e9b4w7e1c", pod="kube-state-metrics-hcp-8654bffc85-2b4bl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss453dqari5t3de9k8dotol85jfinfd-q0j6a1e9b4w7e1c/kube-state-metrics-hcp-8654bffc85-2b4bl has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-16T13:03:28Z
  Ended:       2026-09-16T13:41:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45ahkai3d0q9fcpsr6mjdaf7nr6ha-o2t8d3y0l7d1x7n", pod="kube-state-metrics-hcp-8654bffc85-6v5ws", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45ahkai3d0q9fcpsr6mjdaf7nr6ha-o2t8d3y0l7d1x7n/kube-state-metrics-hcp-8654bffc85-6v5ws has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-16T13:03:28Z
  Ended:       2026-09-16T13:43:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45amlhvl451enbef5jfjbe02g63a8-f1q4q6k8n9u8m3k", pod="kube-state-metrics-hcp-8654bffc85-8r7bj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45amlhvl451enbef5jfjbe02g63a8-f1q4q6k8n9u8m3k/kube-state-metrics-hcp-8654bffc85-8r7bj has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-16T13:03:28Z
  Ended:       2026-09-16T13:32:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45d2ps5pdtsqgj43r9su0i07fh1h8-m2y3w9f0u0r9k3l", pod="kube-state-metrics-hcp-8654bffc85-hzvqb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45d2ps5pdtsqgj43r9su0i07fh1h8-m2y3w9f0u0r9k3l/kube-state-metrics-hcp-8654bffc85-hzvqb has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-16T13:03:28Z
  Ended:       2026-09-16T13:21:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45avj4f66o61uit0kcksueujolvbq-k0e8m9j1h2j9u0i", pod="router-7f899b48d7-n6wr7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45avj4f66o61uit0kcksueujolvbq-k0e8m9j1h2j9u0i/router-7f899b48d7-n6wr7 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:17:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45dhik6j6sb9fbaaodapkvcpnjfmt-f4z1z7c9x4w9k3g", pod="kube-state-metrics-hcp-8654bffc85-wbs5w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45dhik6j6sb9fbaaodapkvcpnjfmt-f4z1z7c9x4w9k3g/kube-state-metrics-hcp-8654bffc85-wbs5w has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:31:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45a0lv485th7ov853bmhqda7d1grb-f5l5n5v0d8y2u6d", pod="kube-state-metrics-hcp-8654bffc85-mpj29", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45a0lv485th7ov853bmhqda7d1grb-f5l5n5v0d8y2u6d/kube-state-metrics-hcp-8654bffc85-mpj29 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:22:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45aneg2j2o4qth6flecfajj5tvki4-l2o8u6v9c5n7n6q", pod="kube-state-metrics-hcp-8654bffc85-pc8f8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45aneg2j2o4qth6flecfajj5tvki4-l2o8u6v9c5n7n6q/kube-state-metrics-hcp-8654bffc85-pc8f8 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:41:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45aj504lvh1njbod15kk0htfcugdd-w0e0y5e7r4m3o8g", pod="kube-state-metrics-hcp-8654bffc85-smbng", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45aj504lvh1njbod15kk0htfcugdd-w0e0y5e7r4m3o8g/kube-state-metrics-hcp-8654bffc85-smbng has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:41:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45alqfdjojoc6uvrrquvn69ep86hp-p4l3h4i3y1n7n9p", pod="kube-state-metrics-hcp-8654bffc85-kts9j", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45alqfdjojoc6uvrrquvn69ep86hp-p4l3h4i3y1n7n9p/kube-state-metrics-hcp-8654bffc85-kts9j has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:17:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45isafobnnvse3kq39umk693uve6h-r2t0l1d7c7a9d1e", pod="kube-state-metrics-hcp-8654bffc85-mvs9w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45isafobnnvse3kq39umk693uve6h-r2t0l1d7c7a9d1e/kube-state-metrics-hcp-8654bffc85-mvs9w has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:27:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45d23ufser4jmn2hba7t0vm2f38rb-g2m8b0v1u1t7v8f", pod="kube-state-metrics-hcp-8654bffc85-gtx4b", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45d23ufser4jmn2hba7t0vm2f38rb-g2m8b0v1u1t7v8f/kube-state-metrics-hcp-8654bffc85-gtx4b has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45ai5u5hobiuu876l5gr5krkhf59n-x0a0u8j1u0z8t2z", pod="kube-state-metrics-hcp-8654bffc85-8xwn8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45ai5u5hobiuu876l5gr5krkhf59n-x0a0u8j1u0z8t2z/kube-state-metrics-hcp-8654bffc85-8xwn8 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:39:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45am3qd3i5mseg9vp5t7veg60i9u8-f9b4r1g2v3v3y5u", pod="kube-state-metrics-hcp-8654bffc85-plqmq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45am3qd3i5mseg9vp5t7veg60i9u8-f9b4r1g2v3v3y5u/kube-state-metrics-hcp-8654bffc85-plqmq has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:11:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45cq3i9jdtaiu413fjc4tb5c4se9l-basic-cluster", pod="kube-state-metrics-hcp-8654bffc85-zm5pl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45cq3i9jdtaiu413fjc4tb5c4se9l-basic-cluster/kube-state-metrics-hcp-8654bffc85-zm5pl has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:43:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45dik9ueoegvhtmnvsnteoq0g4du1-n7b6v4i2o7m1u0u", pod="kube-state-metrics-hcp-8654bffc85-slk75", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45dik9ueoegvhtmnvsnteoq0g4du1-n7b6v4i2o7m1u0u/kube-state-metrics-hcp-8654bffc85-slk75 has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:22:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45aknhr7bgqono4l6o0gk5trcumoj-b3b9b1i0u4d3u6y", pod="kube-state-metrics-hcp-8654bffc85-hqwvp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45aknhr7bgqono4l6o0gk5trcumoj-b3b9b1i0u4d3u6y/kube-state-metrics-hcp-8654bffc85-hqwvp has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss453dfmfp3h1a6k0tbejjbctsua7j9-k1i2q3i9i2w2j2x", pod="kube-state-metrics-hcp-8654bffc85-2jmxg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss453dfmfp3h1a6k0tbejjbctsua7j9-k1i2q3i9i2w2j2x/kube-state-metrics-hcp-8654bffc85-2jmxg has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:34:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45djm06hu474k4gire3e7l44olc34-b3g5s1b1p5g3w0y", pod="kube-state-metrics-hcp-8654bffc85-w7q5h", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45djm06hu474k4gire3e7l44olc34-b3g5s1b1p5g3w0y/kube-state-metrics-hcp-8654bffc85-w7q5h has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:23:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45d4v7s5h34gf677lbu6stjtqtp5c-g5x8y2b6l9p8d1e", pod="kube-state-metrics-hcp-8654bffc85-gkwrf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45d4v7s5h34gf677lbu6stjtqtp5c-g5x8y2b6l9p8d1e/kube-state-metrics-hcp-8654bffc85-gkwrf has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:24:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45alt0da9lepo6tu2pvh5u7s687dp-ephemeral-disk", pod="kube-state-metrics-hcp-8654bffc85-xcpj5", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45alt0da9lepo6tu2pvh5u7s687dp-ephemeral-disk/kube-state-metrics-hcp-8654bffc85-xcpj5 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:21:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45bc09v53lfjcmj3g54u420ob77s8-r0l2r4e5i1s6l2h", pod="router-69b5485d74-72t5c", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45bc09v53lfjcmj3g54u420ob77s8-r0l2r4e5i1s6l2h/router-69b5485d74-72t5c has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-16T13:03:29Z
  Ended:       2026-09-16T13:22:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45d7dagj2q4fc95vdlgrbjcrkagmi-k5b4r3j2v6g5e7k", pod="kube-state-metrics-hcp-8654bffc85-wxmr4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45d7dagj2q4fc95vdlgrbjcrkagmi-k5b4r3j2v6g5e7k/kube-state-metrics-hcp-8654bffc85-wxmr4 has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:19:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss455r3f3n7sks7s8o3chfas3qcq56n-ingress-cert-le", pod="router-764784758f-gqndx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss455r3f3n7sks7s8o3chfas3qcq56n-ingress-cert-le/router-764784758f-gqndx has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:21:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45c06aho04tppkno3kefcqn1a2sjs-ea-listrtdzxt", pod="kube-state-metrics-hcp-8654bffc85-fds47", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45c06aho04tppkno3kefcqn1a2sjs-ea-listrtdzxt/kube-state-metrics-hcp-8654bffc85-fds47 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:24:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45c5gu0g5rk3sue62gv6srpt0sv19-f9y3n5c1f1h3h1e", pod="kube-state-metrics-hcp-8654bffc85-s96gm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45c5gu0g5rk3sue62gv6srpt0sv19-f9y3n5c1f1h3h1e/kube-state-metrics-hcp-8654bffc85-s96gm has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:31:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45al5vb34g2kbgdpjjtt49tus9qu7-cilium-cl", pod="kube-state-metrics-hcp-8654bffc85-7m95x", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45al5vb34g2kbgdpjjtt49tus9qu7-cilium-cl/kube-state-metrics-hcp-8654bffc85-7m95x has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:38:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45d5p3kjg2dkomkt1pvkud0d8fint-k5n3r3f4t4j4d9f", pod="kube-state-metrics-hcp-8654bffc85-5zf8t", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45d5p3kjg2dkomkt1pvkud0d8fint-k5n3r3f4t4j4d9f/kube-state-metrics-hcp-8654bffc85-5zf8t has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:26:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45au25d8jlpb2kbjloe06frtlt4u1-g4f9f6d7f1k2n3j", pod="kube-state-metrics-hcp-8654bffc85-2w2pn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45au25d8jlpb2kbjloe06frtlt4u1-g4f9f6d7f1k2n3j/kube-state-metrics-hcp-8654bffc85-2w2pn has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:20:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss463phv434m12h3cgfclekieomu7eo-cilium-cluster", pod="router-67b67d74d4-4d6mj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss463phv434m12h3cgfclekieomu7eo-cilium-cluster/router-67b67d74d4-4d6mj has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:24:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45d0pa1sd8jpnk5mmr7bmpmc1uh27-d9u9b5p5t5x3a7o", pod="kube-state-metrics-hcp-8654bffc85-jzd7x", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45d0pa1sd8jpnk5mmr7bmpmc1uh27-d9u9b5p5t5x3a7o/kube-state-metrics-hcp-8654bffc85-jzd7x has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:44:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45enm2q1mrr0ikcmo5so09hpk1440-g7s2v6e1t4t3v3w", pod="kube-state-metrics-hcp-8654bffc85-strhx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45enm2q1mrr0ikcmo5so09hpk1440-g7s2v6e1t4t3v3w/kube-state-metrics-hcp-8654bffc85-strhx has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:17:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45alef9infcda8pp6lj56gnqiqkt1-s2m0d6m5z5x1p1o", pod="kube-state-metrics-hcp-8654bffc85-rlzdp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45alef9infcda8pp6lj56gnqiqkt1-s2m0d6m5z5x1p1o/kube-state-metrics-hcp-8654bffc85-rlzdp has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:12:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss45ithac2bg4g097vh3s95jf7bpkvs-s3q1o0z3y5x3y8v", pod="kube-state-metrics-hcp-8654bffc85-rj8kx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss45ithac2bg4g097vh3s95jf7bpkvs-s3q1o0z3y5x3y8v/kube-state-metrics-hcp-8654bffc85-rj8kx has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-09-16T13:04:30Z
  Ended:       2026-09-16T13:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3128064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss457tpj3vtp5gmo526i67mlaqoerts-ea-cluster", pod="kube-state-metrics-hcp-8654bffc85-4cgrf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss457tpj3vtp5gmo526i67mlaqoerts-ea-cluster/kube-state-metrics-hcp-8654bffc85-4cgrf has been in a non-ready state for longer than 5 minutes.

time=2026-09-16T12:29:06.362Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T12:29:07.004Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T12:29:08.123Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=7f3003552f349523d7e6c238bf9cccd3058ec60bc5ec75c8955458c50e5a5797 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5931136-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F7f3003552f349523d7e6c238bf9cccd3058ec60bc5ec75c8955458c50e5a5797
time=2026-09-16T12:59:06.018Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

time=2026-09-16T12:27:55.094Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-09-16T12:27:56.479Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-09-16T12:27:57.833Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=b489aeac3f7b0efc7498443b42a85f2f65ee62ab4f82355239f486d3dbc78483 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5020544-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb489aeac3f7b0efc7498443b42a85f2f65ee62ab4f82355239f486d3dbc78483
time=2026-09-16T12:31:28.399Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5020544-svc/providers/Microsoft.Resources/deployments/b489aeac3f7b0efc7498443b42a85f2f65ee62ab4f82355239f486d3dbc78483/operationStatuses/08584120452084424414\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5020544-svc/providers/Microsoft.Resources/deployments/cluster-6sk23df5akobi\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"BadRequest\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5020544-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j5020544-svc\\\",\\r\\n            \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"GetMSICredentialError\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Get managed identity credential failed. Details: Category: ClientError; Code: NotFound; SubCode: NotFound; Message: unexpected response from MSI data plane, length of returned certificate: 0; Dependency: MSI Credential; AKSTeam: RP\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-16T12:29:10.225Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T12:29:10.864Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-16T12:29:11.770Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=1b97e8958d39ef4f7e29556f7815dac2c009b4e6ab514d8c7781025dc9895978 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6483072-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F1b97e8958d39ef4f7e29556f7815dac2c009b4e6ab514d8c7781025dc9895978
time=2026-09-16T12:59:09.868Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

time=2026-09-16T12:28:37.195Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T12:28:38.137Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T12:28:39.285Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=a97919f9baaee18d96914ec7a53b966faabe4a5aaaf7dd5f49d556a1afbdb617 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2909440-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fa97919f9baaee18d96914ec7a53b966faabe4a5aaaf7dd5f49d556a1afbdb617
time=2026-09-16T12:58:36.820Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0009a0c20>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-xcwwm98r852j" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9e7e240-22cf-4186-a0b5-ed5964184230
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9e7e240-22cf-4186-a0b5-ed5964184230",
      "name": "c9e7e240-22cf-4186-a0b5-ed5964184230",
      "status": "Failed",
      "startTime": "2026-09-16T12:39:41.842014119Z",
      "endTime": "2026-09-16T13:08:44.867300406Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0009a0c20>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-xcwwm98r852j" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9e7e240-22cf-4186-a0b5-ed5964184230
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c9e7e240-22cf-4186-a0b5-ed5964184230",
      "name": "c9e7e240-22cf-4186-a0b5-ed5964184230",
      "status": "Failed",
      "startTime": "2026-09-16T12:39:41.842014119Z",
      "endTime": "2026-09-16T13:08:44.867300406Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-7k4hphh5btnd/cluster-candidate-4-20-f2v4f7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000cf6760>: 
    failed to create HCP cluster cluster-candidate-4-20-f2v4f7: failed waiting for cluster="cluster-candidate-4-20-f2v4f7" in resourcegroup="rg-candidate-4-20-7k4hphh5btnd" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af3ecd66-e6f0-4eb3-95a7-f5a158757dbd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af3ecd66-e6f0-4eb3-95a7-f5a158757dbd",
      "name": "af3ecd66-e6f0-4eb3-95a7-f5a158757dbd",
      "status": "Failed",
      "startTime": "2026-09-16T12:40:01.27881814Z",
      "endTime": "2026-09-16T13:09:04.874482625Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 20m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-7k4hphh5btnd/cluster-candidate-4-20-f2v4f7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000cf6760>: 
    failed to create HCP cluster cluster-candidate-4-20-f2v4f7: failed waiting for cluster="cluster-candidate-4-20-f2v4f7" in resourcegroup="rg-candidate-4-20-7k4hphh5btnd" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af3ecd66-e6f0-4eb3-95a7-f5a158757dbd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af3ecd66-e6f0-4eb3-95a7-f5a158757dbd",
      "name": "af3ecd66-e6f0-4eb3-95a7-f5a158757dbd",
      "status": "Failed",
      "startTime": "2026-09-16T12:40:01.27881814Z",
      "endTime": "2026-09-16T13:09:04.874482625Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 20m32s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0010fab40>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-dztrtj585gtc" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e071038e-8e42-4a4f-90ab-1d7962337b4e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e071038e-8e42-4a4f-90ab-1d7962337b4e",
      "name": "e071038e-8e42-4a4f-90ab-1d7962337b4e",
      "status": "Failed",
      "startTime": "2026-09-16T12:41:22.959566801Z",
      "endTime": "2026-09-16T13:10:24.832743272Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0010fab40>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-dztrtj585gtc" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e071038e-8e42-4a4f-90ab-1d7962337b4e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e071038e-8e42-4a4f-90ab-1d7962337b4e",
      "name": "e071038e-8e42-4a4f-90ab-1d7962337b4e",
      "status": "Failed",
      "startTime": "2026-09-16T12:41:22.959566801Z",
      "endTime": "2026-09-16T13:10:24.832743272Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m16s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c10220>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-89n7j94drhpq" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/494c647c-02a4-4a85-8468-6289598342b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/494c647c-02a4-4a85-8468-6289598342b3",
      "name": "494c647c-02a4-4a85-8468-6289598342b3",
      "status": "Failed",
      "startTime": "2026-09-16T12:39:36.538874537Z",
      "endTime": "2026-09-16T13:08:44.880114684Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c10220>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-89n7j94drhpq" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/494c647c-02a4-4a85-8468-6289598342b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/494c647c-02a4-4a85-8468-6289598342b3",
      "name": "494c647c-02a4-4a85-8468-6289598342b3",
      "status": "Failed",
      "startTime": "2026-09-16T12:39:36.538874537Z",
      "endTime": "2026-09-16T13:08:44.880114684Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m35s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d19280>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-z9lx64t7sbcs" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a0788ae-1307-4f76-a260-f9ac34be043b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a0788ae-1307-4f76-a260-f9ac34be043b",
      "name": "9a0788ae-1307-4f76-a260-f9ac34be043b",
      "status": "Failed",
      "startTime": "2026-09-16T12:39:59.621910893Z",
      "endTime": "2026-09-16T13:09:04.749574347Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d19280>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-z9lx64t7sbcs" to finish creating: GET https://rp.j5673216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a0788ae-1307-4f76-a260-f9ac34be043b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9a0788ae-1307-4f76-a260-f9ac34be043b",
      "name": "9a0788ae-1307-4f76-a260-f9ac34be043b",
      "status": "Failed",
      "startTime": "2026-09-16T12:39:59.621910893Z",
      "endTime": "2026-09-16T13:09:04.749574347Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T12:51:57Z
  Ended:       2026-09-16T13:20:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5673216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss402jtj3okm3l0itnvu8lp11ckp79a-v20261001", pod="router-6c568fff88-cgf48", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss402jtj3okm3l0itnvu8lp11ckp79a-v20261001/router-6c568fff88-cgf48 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--79b927dmgkb5/cluster-hs-pre-b5vpk5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00101e8e0>: 
    failed to create HCP cluster cluster-hs-pre-b5vpk5: failed waiting for cluster="cluster-hs-pre-b5vpk5" in resourcegroup="rg-hs-pre--79b927dmgkb5" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7036fcd-648a-4938-a4d2-03f3c4c3a098
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7036fcd-648a-4938-a4d2-03f3c4c3a098",
      "name": "b7036fcd-648a-4938-a4d2-03f3c4c3a098",
      "status": "Failed",
      "startTime": "2026-09-16T12:41:50.885675594Z",
      "endTime": "2026-09-16T13:10:51.145718175Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--79b927dmgkb5/cluster-hs-pre-b5vpk5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00101e8e0>: 
    failed to create HCP cluster cluster-hs-pre-b5vpk5: failed waiting for cluster="cluster-hs-pre-b5vpk5" in resourcegroup="rg-hs-pre--79b927dmgkb5" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7036fcd-648a-4938-a4d2-03f3c4c3a098
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7036fcd-648a-4938-a4d2-03f3c4c3a098",
      "name": "b7036fcd-648a-4938-a4d2-03f3c4c3a098",
      "status": "Failed",
      "startTime": "2026-09-16T12:41:50.885675594Z",
      "endTime": "2026-09-16T13:10:51.145718175Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000706aa0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dwmf5wnrnqjn" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7362da66-e5d5-454e-b456-4b2dce3364f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7362da66-e5d5-454e-b456-4b2dce3364f4",
      "name": "7362da66-e5d5-454e-b456-4b2dce3364f4",
      "status": "Failed",
      "startTime": "2026-09-16T12:42:20.52077321Z",
      "endTime": "2026-09-16T13:11:21.200973872Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000706aa0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dwmf5wnrnqjn" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7362da66-e5d5-454e-b456-4b2dce3364f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7362da66-e5d5-454e-b456-4b2dce3364f4",
      "name": "7362da66-e5d5-454e-b456-4b2dce3364f4",
      "status": "Failed",
      "startTime": "2026-09-16T12:42:20.52077321Z",
      "endTime": "2026-09-16T13:11:21.200973872Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000af3c40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hvkg6dzj54xx" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f44314ae-97e1-419e-bb21-2e04874f19c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f44314ae-97e1-419e-bb21-2e04874f19c3",
      "name": "f44314ae-97e1-419e-bb21-2e04874f19c3",
      "status": "Failed",
      "startTime": "2026-09-16T12:42:40.715298605Z",
      "endTime": "2026-09-16T13:11:41.209044126Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000af3c40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hvkg6dzj54xx" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f44314ae-97e1-419e-bb21-2e04874f19c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f44314ae-97e1-419e-bb21-2e04874f19c3",
      "name": "f44314ae-97e1-419e-bb21-2e04874f19c3",
      "status": "Failed",
      "startTime": "2026-09-16T12:42:40.715298605Z",
      "endTime": "2026-09-16T13:11:41.209044126Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-crjc9ggv5c5c/cluster-candidate-4-22-kthmtb should provision
Unexpected error:
    <*fmt.wrapError | 0xc001180160>: 
    failed to create HCP cluster cluster-candidate-4-22-kthmtb: failed waiting for cluster="cluster-candidate-4-22-kthmtb" in resourcegroup="rg-candidate-4-22-crjc9ggv5c5c" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c590b0b-f53f-4d94-8c31-965c6434354b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c590b0b-f53f-4d94-8c31-965c6434354b",
      "name": "9c590b0b-f53f-4d94-8c31-965c6434354b",
      "status": "Failed",
      "startTime": "2026-09-16T12:42:02.207518072Z",
      "endTime": "2026-09-16T13:11:11.226520946Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m34s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-crjc9ggv5c5c/cluster-candidate-4-22-kthmtb should provision
Unexpected error:
    <*fmt.wrapError | 0xc001180160>: 
    failed to create HCP cluster cluster-candidate-4-22-kthmtb: failed waiting for cluster="cluster-candidate-4-22-kthmtb" in resourcegroup="rg-candidate-4-22-crjc9ggv5c5c" to finish creating: GET https://rp.j6923776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c590b0b-f53f-4d94-8c31-965c6434354b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9c590b0b-f53f-4d94-8c31-965c6434354b",
      "name": "9c590b0b-f53f-4d94-8c31-965c6434354b",
      "status": "Failed",
      "startTime": "2026-09-16T12:42:02.207518072Z",
      "endTime": "2026-09-16T13:11:11.226520946Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m34s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T12:57:26Z
  Ended:       2026-09-16T13:17:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6923776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss41gibi7sp422ii8he0pq63nn52g37-private-kas", pod="router-6b58c6bb5b-92m4m", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss41gibi7sp422ii8he0pq63nn52g37-private-kas/router-6b58c6bb5b-92m4m has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T12:57:26Z
  Ended:       2026-09-16T13:17:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6923776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss41b4su9bid10hrqo3uiiuf5p3hgs4-b8r7o4e7s7g6i3a", pod="router-b8d96bd58-thzdn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss41b4su9bid10hrqo3uiiuf5p3hgs4-b8r7o4e7s7g6i3a/router-b8d96bd58-thzdn has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T12:57:26Z
  Ended:       2026-09-16T13:19:25Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6923776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss416rp6lv3sb7n8d5pdpc3a57h0rqp-j4u9m8f9s6l0v4z", pod="router-8fbb8c5f8-swtx6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss416rp6lv3sb7n8d5pdpc3a57h0rqp-j4u9m8f9s6l0v4z/router-8fbb8c5f8-swtx6 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc001a980c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-gztjzzfvghvl" to finish deleting: GET https://rp.j1755904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/067d2d4c-07a0-4859-8081-90eb78e5b132
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/067d2d4c-07a0-4859-8081-90eb78e5b132",
      "name": "067d2d4c-07a0-4859-8081-90eb78e5b132",
      "status": "Failed",
      "startTime": "2026-09-16T12:52:08.640021777Z",
      "endTime": "2026-09-16T13:16:14.927620259Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss3v2g1581u9fmlnc5tsgv7vcs4icad still exists (deletion dispatched at 2026-09-16T12:52:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc001a980c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-gztjzzfvghvl" to finish deleting: GET https://rp.j1755904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/067d2d4c-07a0-4859-8081-90eb78e5b132
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/067d2d4c-07a0-4859-8081-90eb78e5b132",
      "name": "067d2d4c-07a0-4859-8081-90eb78e5b132",
      "status": "Failed",
      "startTime": "2026-09-16T12:52:08.640021777Z",
      "endTime": "2026-09-16T13:16:14.927620259Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss3v2g1581u9fmlnc5tsgv7vcs4icad still exists (deletion dispatched at 2026-09-16T12:52:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc0013980d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-9gp2cv4556mc" to finish deleting: GET https://rp.j1755904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8e89b79-b819-4d57-beb6-45d5fd38c76b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8e89b79-b819-4d57-beb6-45d5fd38c76b",
      "name": "f8e89b79-b819-4d57-beb6-45d5fd38c76b",
      "status": "Failed",
      "startTime": "2026-09-16T12:52:15.549173368Z",
      "endTime": "2026-09-16T13:16:24.936632758Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss3unt2thcjv9iur8dobv2ob7ghr15h still exists (deletion dispatched at 2026-09-16T12:52:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc0013980d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-9gp2cv4556mc" to finish deleting: GET https://rp.j1755904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8e89b79-b819-4d57-beb6-45d5fd38c76b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f8e89b79-b819-4d57-beb6-45d5fd38c76b",
      "name": "f8e89b79-b819-4d57-beb6-45d5fd38c76b",
      "status": "Failed",
      "startTime": "2026-09-16T12:52:15.549173368Z",
      "endTime": "2026-09-16T13:16:24.936632758Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss3unt2thcjv9iur8dobv2ob7ghr15h still exists (deletion dispatched at 2026-09-16T12:52:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc0017780a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-225ts6vlpjr6" to finish deleting: GET https://rp.j1755904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2359d86d-b689-4fd0-915b-2d216b861e83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2359d86d-b689-4fd0-915b-2d216b861e83",
      "name": "2359d86d-b689-4fd0-915b-2d216b861e83",
      "status": "Failed",
      "startTime": "2026-09-16T12:52:03.476346261Z",
      "endTime": "2026-09-16T13:16:04.956406239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss3uucne5o52iff154le845qrjeepnu still exists (deletion dispatched at 2026-09-16T12:52:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error:
    <*errors.joinError | 0xc0017780a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-225ts6vlpjr6" to finish deleting: GET https://rp.j1755904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2359d86d-b689-4fd0-915b-2d216b861e83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2359d86d-b689-4fd0-915b-2d216b861e83",
      "name": "2359d86d-b689-4fd0-915b-2d216b861e83",
      "status": "Failed",
      "startTime": "2026-09-16T12:52:03.476346261Z",
      "endTime": "2026-09-16T13:16:04.956406239Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss3uucne5o52iff154le845qrjeepnu still exists (deletion dispatched at 2026-09-16T12:52:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.093s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000ff3040>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/volumeattachments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiextensions.k8s.io/customresourcedefinitions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list policy/poddisruptionbudgets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list autoscaling/horizontalpodautoscalers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list coordination.k8s.io/leases: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list discovery.k8s.io/endpointslices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list certificates.k8s.io/certificatesigningrequests: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list scheduling.k8s.io/priorityclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list node.k8s.io/runtimeclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/mutatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/validatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.093s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000ff3040>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/volumeattachments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiextensions.k8s.io/customresourcedefinitions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list policy/poddisruptionbudgets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list autoscaling/horizontalpodautoscalers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list coordination.k8s.io/leases: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list discovery.k8s.io/endpointslices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list certificates.k8s.io/certificatesigningrequests: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list scheduling.k8s.io/priorityclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list node.k8s.io/runtimeclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/mutatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/validatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc001092780>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jqqcvh44krzk" to finish creating: GET https://rp.j0897152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad8bf203-db0b-47d7-bd3b-a158a50c55a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad8bf203-db0b-47d7-bd3b-a158a50c55a1",
      "name": "ad8bf203-db0b-47d7-bd3b-a158a50c55a1",
      "status": "Failed",
      "startTime": "2026-09-16T12:17:03.393436374Z",
      "endTime": "2026-09-16T12:46:10.759018774Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc001092780>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-jqqcvh44krzk" to finish creating: GET https://rp.j0897152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad8bf203-db0b-47d7-bd3b-a158a50c55a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ad8bf203-db0b-47d7-bd3b-a158a50c55a1",
      "name": "ad8bf203-db0b-47d7-bd3b-a158a50c55a1",
      "status": "Failed",
      "startTime": "2026-09-16T12:17:03.393436374Z",
      "endTime": "2026-09-16T12:46:10.759018774Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T12:49:06Z
  Ended:       2026-09-16T13:01:06Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0897152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.188:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0897152/providers/microsoft.monitor/accounts/hcps-j0897152", namespace="ocm-arohcpci01-2ss3lg4qnv008n1sgqgucdnh6ucncan0-j1s9x0r6b8u1v3h", pod="arohcp-monitor-kube-state-metrics-574f657d96-8rm4f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss3lg4qnv008n1sgqgucdnh6ucncan0-j1s9x0r6b8u1v3h/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T12:49:07Z
  Ended:       2026-09-16T12:56:03Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0897152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.188:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0897152/providers/microsoft.monitor/accounts/hcps-j0897152", namespace="ocm-arohcpci01-2ss3lg4qnv008n1sgqgucdnh6ucncan0-j1s9x0r6b8u1v3h", pod="arohcp-monitor-kube-state-metrics-574f657d96-8rm4f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss3lg4qnv008n1sgqgucdnh6ucncan0-j1s9x0r6b8u1v3h/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T12:47:51Z
  Ended:       2026-09-16T13:06:54Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0897152-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-lswwb", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-lswwb (namespace arobit) on cluster ci01-j0897152-mgmt-2 has averaged 174.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-16T12:47:51Z
  Ended:       2026-09-16T13:04:57Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0897152-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-664vn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-664vn (namespace arobit) on cluster ci01-j0897152-svc has averaged 169.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-16T12:47:51Z
  Ended:       2026-09-16T13:10:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0897152-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-84794bbc8-5fmm4", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-84794bbc8-5fmm4 (namespace clusters-service) on cluster ci01-j0897152-svc has averaged 218.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T12:47:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0897152-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-655696f65-mr299", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-655696f65-mr299 (namespace aro-hcp) on cluster ci01-j0897152-svc has averaged 400.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-16T12:47:51Z
  Ended:       2026-09-16T12:55:52Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0897152-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-84794bbc8-wt8kz", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-84794bbc8-wt8kz (namespace clusters-service) on cluster ci01-j0897152-svc has averaged 759.8% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-16T12:54:52Z
  Ended:       2026-09-16T13:05:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j0897152-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-cbmhn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-cbmhn (namespace arobit) on cluster ci01-j0897152-mgmt-1 has averaged 131.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001ff26c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-g5s8ftp9hh9t" to finish creating: GET https://rp.j4241024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "name": "06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "status": "Failed",
      "startTime": "2026-09-16T12:18:53.259035342Z",
      "endTime": "2026-09-16T12:37:53.675934016Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001ff26c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-g5s8ftp9hh9t" to finish creating: GET https://rp.j4241024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "name": "06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "status": "Failed",
      "startTime": "2026-09-16T12:18:53.259035342Z",
      "endTime": "2026-09-16T12:37:53.675934016Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 15 time(s) (4 unknown, 11 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T12:06:16Z
  Ended:       2026-09-16T12:16:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6486656-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss3bi7r1h4v48sj30qba1fo8pvrqm7f-l6r4d7v5o3c4z2a", pod="kube-apiserver-854cf58859-z8glp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss3bi7r1h4v48sj30qba1fo8pvrqm7f-l6r4d7v5o3c4z2a/kube-apiserver-854cf58859-z8glp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T12:07:18Z
  Ended:       2026-09-16T12:17:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6486656-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss3b4g6moo56neqpa7uf1if0ci5u32q-k5p9e7w4i8s6c1y", pod="kube-apiserver-76cdc4f787-whddh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss3b4g6moo56neqpa7uf1if0ci5u32q-k5p9e7w4i8s6c1y/kube-apiserver-76cdc4f787-whddh has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T12:07:18Z
  Ended:       2026-09-16T12:17:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6486656-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss3b570knsufmrkkurvp62eg2ftj2qm-e6i6j3u6a5v6u0v", pod="etcd-0", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss3b570knsufmrkkurvp62eg2ftj2qm-e6i6j3u6a5v6u0v/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T12:07:18Z
  Ended:       2026-09-16T12:16:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6486656-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss3b570knsufmrkkurvp62eg2ftj2qm-e6i6j3u6a5v6u0v", pod="ignition-server-79c7574ccc-qw8dp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss3b570knsufmrkkurvp62eg2ftj2qm-e6i6j3u6a5v6u0v/ignition-server-79c7574ccc-qw8dp has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00019f020>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-t55rrq" in resourcegroup="rg-np-skip-minor-t55rrq-wdl65v69n5wm" to finish creating: GET https://rp.j1102208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0da7c7c-5ccf-448f-9d31-2330d22070c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0da7c7c-5ccf-448f-9d31-2330d22070c5",
      "name": "a0da7c7c-5ccf-448f-9d31-2330d22070c5",
      "status": "Failed",
      "startTime": "2026-09-16T11:32:30.704689728Z",
      "endTime": "2026-09-16T11:51:34.799712169Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00019f020>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-t55rrq" in resourcegroup="rg-np-skip-minor-t55rrq-wdl65v69n5wm" to finish creating: GET https://rp.j1102208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0da7c7c-5ccf-448f-9d31-2330d22070c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0da7c7c-5ccf-448f-9d31-2330d22070c5",
      "name": "a0da7c7c-5ccf-448f-9d31-2330d22070c5",
      "status": "Failed",
      "startTime": "2026-09-16T11:32:30.704689728Z",
      "endTime": "2026-09-16T11:51:34.799712169Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-16T10:50:03.391Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T10:50:04.001Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T10:50:05.394Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=39c0bfe6f31a9b99040636b0dd2da01bd62593c0723e012a25f7c174f8b2444b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5324416-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F39c0bfe6f31a9b99040636b0dd2da01bd62593c0723e012a25f7c174f8b2444b
time=2026-09-16T11:20:03.104Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:111]: failed to create node pool "np-w5pk5g" for cluster "cluster-hs-pre-w5pk5g"
Unexpected error:
    <*fmt.wrapError | 0xc00108f320>: 
    failed to create NodePool np-w5pk5g: failed waiting for nodepool="np-w5pk5g" for cluster "cluster-hs-pre-w5pk5g" in resourcegroup="rg-hs-pre--tkbl565tmmql" to finish creating: GET https://rp.j6364288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2191135-7ef2-4813-a443-f5ccab881558
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2191135-7ef2-4813-a443-f5ccab881558",
      "name": "f2191135-7ef2-4813-a443-f5ccab881558",
      "status": "Failed",
      "startTime": "2026-09-16T11:03:25.0216776Z",
      "endTime": "2026-09-16T11:22:33.00315027Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:111]: failed to create node pool "np-w5pk5g" for cluster "cluster-hs-pre-w5pk5g"
Unexpected error:
    <*fmt.wrapError | 0xc00108f320>: 
    failed to create NodePool np-w5pk5g: failed waiting for nodepool="np-w5pk5g" for cluster "cluster-hs-pre-w5pk5g" in resourcegroup="rg-hs-pre--tkbl565tmmql" to finish creating: GET https://rp.j6364288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2191135-7ef2-4813-a443-f5ccab881558
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2191135-7ef2-4813-a443-f5ccab881558",
      "name": "f2191135-7ef2-4813-a443-f5ccab881558",
      "status": "Failed",
      "startTime": "2026-09-16T11:03:25.0216776Z",
      "endTime": "2026-09-16T11:22:33.00315027Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.000s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.000s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00230a300>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ql5j7w6fx6vn" to finish creating: GET https://rp.j6364288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728",
      "name": "5fde05ba-68c2-437a-84ea-82765b356728",
      "status": "Failed",
      "startTime": "2026-09-16T11:04:09.351110934Z",
      "endTime": "2026-09-16T11:04:15.302518412Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00230a300>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ql5j7w6fx6vn" to finish creating: GET https://rp.j6364288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728",
      "name": "5fde05ba-68c2-437a-84ea-82765b356728",
      "status": "Failed",
      "startTime": "2026-09-16T11:04:09.351110934Z",
      "endTime": "2026-09-16T11:04:15.302518412Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--9b4s75wtf446/cluster-hs-pre-vpfc9j should provision
Unexpected error:
    <*fmt.wrapError | 0xc0001505c0>: 
    failed to create HCP cluster cluster-hs-pre-vpfc9j: failed waiting for cluster="cluster-hs-pre-vpfc9j" in resourcegroup="rg-hs-pre--9b4s75wtf446" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/be3e1cc0-6007-4f70-bd41-a633b0402774
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/be3e1cc0-6007-4f70-bd41-a633b0402774",
      "name": "be3e1cc0-6007-4f70-bd41-a633b0402774",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:03.039188877Z",
      "endTime": "2026-09-16T11:21:09.718773632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--9b4s75wtf446/cluster-hs-pre-vpfc9j should provision
Unexpected error:
    <*fmt.wrapError | 0xc0001505c0>: 
    failed to create HCP cluster cluster-hs-pre-vpfc9j: failed waiting for cluster="cluster-hs-pre-vpfc9j" in resourcegroup="rg-hs-pre--9b4s75wtf446" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/be3e1cc0-6007-4f70-bd41-a633b0402774
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/be3e1cc0-6007-4f70-bd41-a633b0402774",
      "name": "be3e1cc0-6007-4f70-bd41-a633b0402774",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:03.039188877Z",
      "endTime": "2026-09-16T11:21:09.718773632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000db6000>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xn5sprrvmhpp" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7e44efb7-2fb2-4dd3-851e-724f837768e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7e44efb7-2fb2-4dd3-851e-724f837768e3",
      "name": "7e44efb7-2fb2-4dd3-851e-724f837768e3",
      "status": "Failed",
      "startTime": "2026-09-16T11:10:24.772921169Z",
      "endTime": "2026-09-16T11:29:29.692653081Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000db6000>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xn5sprrvmhpp" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7e44efb7-2fb2-4dd3-851e-724f837768e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7e44efb7-2fb2-4dd3-851e-724f837768e3",
      "name": "7e44efb7-2fb2-4dd3-851e-724f837768e3",
      "status": "Failed",
      "startTime": "2026-09-16T11:10:24.772921169Z",
      "endTime": "2026-09-16T11:29:29.692653081Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00108c078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mzfrqt8tf9wq" to finish deleting: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "name": "e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "status": "Failed",
      "startTime": "2026-09-16T11:18:42.744808437Z",
      "endTime": "2026-09-16T11:42:49.617012035Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00108c078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mzfrqt8tf9wq" to finish deleting: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "name": "e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "status": "Failed",
      "startTime": "2026-09-16T11:18:42.744808437Z",
      "endTime": "2026-09-16T11:42:49.617012035Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0011044c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-95l622wn4sm9" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "name": "1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.869364535Z",
      "endTime": "2026-09-16T11:21:09.719035401Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0011044c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-95l622wn4sm9" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "name": "1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.869364535Z",
      "endTime": "2026-09-16T11:21:09.719035401Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vm24lv8gf4qb/cluster-candidate-4-21-llvd77 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011b0380>: 
    failed to create HCP cluster cluster-candidate-4-21-llvd77: failed waiting for cluster="cluster-candidate-4-21-llvd77" in resourcegroup="rg-candidate-4-21-vm24lv8gf4qb" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "name": "a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.803955113Z",
      "endTime": "2026-09-16T11:21:09.695382854Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vm24lv8gf4qb/cluster-candidate-4-21-llvd77 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011b0380>: 
    failed to create HCP cluster cluster-candidate-4-21-llvd77: failed waiting for cluster="cluster-candidate-4-21-llvd77" in resourcegroup="rg-candidate-4-21-vm24lv8gf4qb" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "name": "a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.803955113Z",
      "endTime": "2026-09-16T11:21:09.695382854Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000aaaa60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6mp8vkxcbh5s" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "name": "d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "status": "Failed",
      "startTime": "2026-09-16T10:51:43.295429479Z",
      "endTime": "2026-09-16T11:20:49.710317459Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000aaaa60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6mp8vkxcbh5s" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "name": "d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "status": "Failed",
      "startTime": "2026-09-16T10:51:43.295429479Z",
      "endTime": "2026-09-16T11:20:49.710317459Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc001082360>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-mqnmkb84gsqq" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86c332b-3661-447d-86c0-3d610ff0d6c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86c332b-3661-447d-86c0-3d610ff0d6c0",
      "name": "a86c332b-3661-447d-86c0-3d610ff0d6c0",
      "status": "Failed",
      "startTime": "2026-09-16T10:51:52.760340073Z",
      "endTime": "2026-09-16T11:20:59.608470437Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:98]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc001082360>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-mqnmkb84gsqq" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86c332b-3661-447d-86c0-3d610ff0d6c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a86c332b-3661-447d-86c0-3d610ff0d6c0",
      "name": "a86c332b-3661-447d-86c0-3d610ff0d6c0",
      "status": "Failed",
      "startTime": "2026-09-16T10:51:52.760340073Z",
      "endTime": "2026-09-16T11:20:59.608470437Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-16T10:35:02.273Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T10:35:02.716Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6352640-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6352640-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:ad132446-7529-417f-aff8-31d826ffb6f2 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#ad132446-7529-417f-aff8-31d826ffb6f2 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T10:35:02.716Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:35:02.863Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T10:35:03.042Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T10:35:03.042Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:35:03.143Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:45:05.949Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:45:05.949Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:46:05.955Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T10:46:06.637Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6352640-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6352640-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:ad132446-7529-417f-aff8-31d826ffb6f2 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#ad132446-7529-417f-aff8-31d826ffb6f2 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T10:46:06.637Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:07.013Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T10:46:07.408Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:07.408Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:07.670Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T10:46:08.488Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:08.489Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:08.888Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:09.401Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T10:46:09.747Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T10:46:09.959Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:10.361Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:10.729Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:11.065Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T10:46:11.065Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:11.148Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T10:56:12.973Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:56:12.973Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:57:12.977Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T10:57:13.486Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6352640-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6352640-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:ad132446-7529-417f-aff8-31d826ffb6f2 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#ad132446-7529-417f-aff8-31d826ffb6f2 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T10:57:13.486Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:13.807Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T10:57:14.174Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:14.174Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:14.404Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T10:57:15.329Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:15.329Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:15.688Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:15.987Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T10:57:16.345Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T10:57:16.535Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:16.747Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:16.984Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:17.208Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T10:57:17.208Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:17.300Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T11:05:02.358Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-16T11:05:02.359Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:41:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6352640-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-4526h", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-4526h has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:41:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6352640-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-q59vx", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-q59vx has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:41:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6352640-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-h4n49", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-h4n49 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:41:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6352640-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-6rpq8", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-6rpq8 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:51:08Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j6352640-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j6352640-svc has been unreachable for 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:51:25Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6352640-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.74:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6352640/providers/microsoft.monitor/accounts/services-j6352640", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-4mlt5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:51:25Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6352640-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.74:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6352640/providers/microsoft.monitor/accounts/services-j6352640", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-4mlt5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-cw6cmg9l2kxw/cluster-candidate-4-22-7hbdnb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000599580>: 
    failed to create HCP cluster cluster-candidate-4-22-7hbdnb: failed waiting for cluster="cluster-candidate-4-22-7hbdnb" in resourcegroup="rg-candidate-4-22-cw6cmg9l2kxw" to finish creating: GET https://rp.j5443840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7411591-3119-4eb0-9f0c-3b28443fa9a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7411591-3119-4eb0-9f0c-3b28443fa9a7",
      "name": "b7411591-3119-4eb0-9f0c-3b28443fa9a7",
      "status": "Failed",
      "startTime": "2026-09-16T10:44:05.206761929Z",
      "endTime": "2026-09-16T11:13:07.811131396Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-cw6cmg9l2kxw/cluster-candidate-4-22-7hbdnb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000599580>: 
    failed to create HCP cluster cluster-candidate-4-22-7hbdnb: failed waiting for cluster="cluster-candidate-4-22-7hbdnb" in resourcegroup="rg-candidate-4-22-cw6cmg9l2kxw" to finish creating: GET https://rp.j5443840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7411591-3119-4eb0-9f0c-3b28443fa9a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b7411591-3119-4eb0-9f0c-3b28443fa9a7",
      "name": "b7411591-3119-4eb0-9f0c-3b28443fa9a7",
      "status": "Failed",
      "startTime": "2026-09-16T10:44:05.206761929Z",
      "endTime": "2026-09-16T11:13:07.811131396Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--9f6m5qx9hqkh/cluster-hs-pre-42s85j should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c9e460>: 
    failed to create HCP cluster cluster-hs-pre-42s85j: failed waiting for cluster="cluster-hs-pre-42s85j" in resourcegroup="rg-hs-pre--9f6m5qx9hqkh" to finish creating: GET https://rp.j5443840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90bc1c83-43b8-4ba3-ab12-9113f5b59f30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90bc1c83-43b8-4ba3-ab12-9113f5b59f30",
      "name": "90bc1c83-43b8-4ba3-ab12-9113f5b59f30",
      "status": "Failed",
      "startTime": "2026-09-16T10:44:25.183290241Z",
      "endTime": "2026-09-16T11:13:27.895359388Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m51s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--9f6m5qx9hqkh/cluster-hs-pre-42s85j should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c9e460>: 
    failed to create HCP cluster cluster-hs-pre-42s85j: failed waiting for cluster="cluster-hs-pre-42s85j" in resourcegroup="rg-hs-pre--9f6m5qx9hqkh" to finish creating: GET https://rp.j5443840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90bc1c83-43b8-4ba3-ab12-9113f5b59f30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/90bc1c83-43b8-4ba3-ab12-9113f5b59f30",
      "name": "90bc1c83-43b8-4ba3-ab12-9113f5b59f30",
      "status": "Failed",
      "startTime": "2026-09-16T10:44:25.183290241Z",
      "endTime": "2026-09-16T11:13:27.895359388Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m51s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s) (1 unknown, 2 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T10:58:27Z
  Ended:       2026-09-16T11:25:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5443840-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss2a2d61qlifdnfp6cit52qi450302e-j1l3m8c5u1f8u2n", pod="router-6d5bd595f9-fx5mz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss2a2d61qlifdnfp6cit52qi450302e-j1l3m8c5u1f8u2n/router-6d5bd595f9-fx5mz has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:185]: failed to create SRE breakglass credentials with aro-sre-pso permissions
Unexpected error:
    <*fmt.wrapError | 0xc000a7f820>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-wlfqkcdjh9rm/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-lbdxk/kubeconfig: timeout waiting for session to become ready (last status: {"HostedControlPlaneAvailable":{"message":"HostedControlPlane exists but is not ready","reason":"HostedControlPlaneNotReady","status":"False"},"status":"Session is not ready"})
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:185]: failed to create SRE breakglass credentials with aro-sre-pso permissions
Unexpected error:
    <*fmt.wrapError | 0xc000a7f820>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-wlfqkcdjh9rm/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-lbdxk/kubeconfig: timeout waiting for session to become ready (last status: {"HostedControlPlaneAvailable":{"message":"HostedControlPlane exists but is not ready","reason":"HostedControlPlaneNotReady","status":"False"},"status":"Session is not ready"})
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: failed to create HCP cluster "cluster-zstream-4-22-kbmqc2" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00111c540>: 
    failed to create HCP cluster cluster-zstream-4-22-kbmqc2: failed waiting for cluster="cluster-zstream-4-22-kbmqc2" in resourcegroup="rg-zstream-upgrade-4-22-plv5jd5xhkrz" to finish creating: GET https://rp.j4184576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54880953-0445-4405-8334-2c8fc8586a9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54880953-0445-4405-8334-2c8fc8586a9a",
      "name": "54880953-0445-4405-8334-2c8fc8586a9a",
      "status": "Failed",
      "startTime": "2026-09-16T10:04:06.201723402Z",
      "endTime": "2026-09-16T10:33:14.400302806Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.13 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: failed to create HCP cluster "cluster-zstream-4-22-kbmqc2" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00111c540>: 
    failed to create HCP cluster cluster-zstream-4-22-kbmqc2: failed waiting for cluster="cluster-zstream-4-22-kbmqc2" in resourcegroup="rg-zstream-upgrade-4-22-plv5jd5xhkrz" to finish creating: GET https://rp.j4184576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54880953-0445-4405-8334-2c8fc8586a9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54880953-0445-4405-8334-2c8fc8586a9a",
      "name": "54880953-0445-4405-8334-2c8fc8586a9a",
      "status": "Failed",
      "startTime": "2026-09-16T10:04:06.201723402Z",
      "endTime": "2026-09-16T10:33:14.400302806Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.13 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T10:17:51Z
  Ended:       2026-09-16T10:41:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4184576-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss1n67ctv55qmobaikqm5f0c9f5u0la-q5h9a2h5l8m7j8d", pod="router-595c8dc9bc-fnrsm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss1n67ctv55qmobaikqm5f0c9f5u0la-q5h9a2h5l8m7j8d/router-595c8dc9bc-fnrsm has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T10:20:03Z
  Ended:       2026-09-16T10:35:01Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4184576-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc001110400>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4p7ps7sph7f6" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e98671-810b-4be1-b0de-33ed4ed7b43e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e98671-810b-4be1-b0de-33ed4ed7b43e",
      "name": "64e98671-810b-4be1-b0de-33ed4ed7b43e",
      "status": "Failed",
      "startTime": "2026-09-16T09:51:36.009109937Z",
      "endTime": "2026-09-16T10:20:38.775019606Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc001110400>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4p7ps7sph7f6" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e98671-810b-4be1-b0de-33ed4ed7b43e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/64e98671-810b-4be1-b0de-33ed4ed7b43e",
      "name": "64e98671-810b-4be1-b0de-33ed4ed7b43e",
      "status": "Failed",
      "startTime": "2026-09-16T09:51:36.009109937Z",
      "endTime": "2026-09-16T10:20:38.775019606Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000ac4380>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g8hsmlndk2gz" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02861d9b-c33b-4f27-b90a-22b7a7e4608e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02861d9b-c33b-4f27-b90a-22b7a7e4608e",
      "name": "02861d9b-c33b-4f27-b90a-22b7a7e4608e",
      "status": "Failed",
      "startTime": "2026-09-16T09:50:12.871173483Z",
      "endTime": "2026-09-16T10:19:18.822990124Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000ac4380>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g8hsmlndk2gz" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02861d9b-c33b-4f27-b90a-22b7a7e4608e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/02861d9b-c33b-4f27-b90a-22b7a7e4608e",
      "name": "02861d9b-c33b-4f27-b90a-22b7a7e4608e",
      "status": "Failed",
      "startTime": "2026-09-16T09:50:12.871173483Z",
      "endTime": "2026-09-16T10:19:18.822990124Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b242a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-zvr2lfbsdzcr" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00bdd26c-d6ec-4e17-85fd-75e0475b1ea2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00bdd26c-d6ec-4e17-85fd-75e0475b1ea2",
      "name": "00bdd26c-d6ec-4e17-85fd-75e0475b1ea2",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:28.418802254Z",
      "endTime": "2026-09-16T10:18:28.723004291Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m29s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b242a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-zvr2lfbsdzcr" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00bdd26c-d6ec-4e17-85fd-75e0475b1ea2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00bdd26c-d6ec-4e17-85fd-75e0475b1ea2",
      "name": "00bdd26c-d6ec-4e17-85fd-75e0475b1ea2",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:28.418802254Z",
      "endTime": "2026-09-16T10:18:28.723004291Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m29s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000830a80>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-v7jwmpn72lgg" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb850b73-14fc-422c-9a1f-30bf8f513961
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb850b73-14fc-422c-9a1f-30bf8f513961",
      "name": "bb850b73-14fc-422c-9a1f-30bf8f513961",
      "status": "Failed",
      "startTime": "2026-09-16T09:50:53.773970906Z",
      "endTime": "2026-09-16T10:19:58.777848052Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000830a80>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-v7jwmpn72lgg" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb850b73-14fc-422c-9a1f-30bf8f513961
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bb850b73-14fc-422c-9a1f-30bf8f513961",
      "name": "bb850b73-14fc-422c-9a1f-30bf8f513961",
      "status": "Failed",
      "startTime": "2026-09-16T09:50:53.773970906Z",
      "endTime": "2026-09-16T10:19:58.777848052Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m42s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--kj428xsfb48q/cluster-hs-pre-bf84l4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00115c700>: 
    failed to create HCP cluster cluster-hs-pre-bf84l4: failed waiting for cluster="cluster-hs-pre-bf84l4" in resourcegroup="rg-hs-pre--kj428xsfb48q" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aea5d780-ec83-4b09-8960-8b31d26d83b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aea5d780-ec83-4b09-8960-8b31d26d83b1",
      "name": "aea5d780-ec83-4b09-8960-8b31d26d83b1",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:11.962303449Z",
      "endTime": "2026-09-16T10:18:18.875942327Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--kj428xsfb48q/cluster-hs-pre-bf84l4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00115c700>: 
    failed to create HCP cluster cluster-hs-pre-bf84l4: failed waiting for cluster="cluster-hs-pre-bf84l4" in resourcegroup="rg-hs-pre--kj428xsfb48q" to finish creating: GET https://rp.j8823168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aea5d780-ec83-4b09-8960-8b31d26d83b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/aea5d780-ec83-4b09-8960-8b31d26d83b1",
      "name": "aea5d780-ec83-4b09-8960-8b31d26d83b1",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:11.962303449Z",
      "endTime": "2026-09-16T10:18:18.875942327Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m21s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00168c138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-px8wfxs26nnj" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cd68e491-5165-4d43-8cd3-0c235f3ba9c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cd68e491-5165-4d43-8cd3-0c235f3ba9c5",
      "name": "cd68e491-5165-4d43-8cd3-0c235f3ba9c5",
      "status": "Failed",
      "startTime": "2026-09-16T09:40:28.518309442Z",
      "endTime": "2026-09-16T10:04:32.390225559Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00168c138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-px8wfxs26nnj" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cd68e491-5165-4d43-8cd3-0c235f3ba9c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cd68e491-5165-4d43-8cd3-0c235f3ba9c5",
      "name": "cd68e491-5165-4d43-8cd3-0c235f3ba9c5",
      "status": "Failed",
      "startTime": "2026-09-16T09:40:28.518309442Z",
      "endTime": "2026-09-16T10:04:32.390225559Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-9bqqftbhhkhb/cluster-candidate-5-0-rgjngf should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f68580>: 
    failed to create HCP cluster cluster-candidate-5-0-rgjngf: failed waiting for cluster="cluster-candidate-5-0-rgjngf" in resourcegroup="rg-candidate-5-0-9bqqftbhhkhb" to finish creating: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3baf4e11-4b0e-4700-b113-da3918d872d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3baf4e11-4b0e-4700-b113-da3918d872d5",
      "name": "3baf4e11-4b0e-4700-b113-da3918d872d5",
      "status": "Failed",
      "startTime": "2026-09-16T09:21:27.799687923Z",
      "endTime": "2026-09-16T09:50:32.506640388Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 18m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-9bqqftbhhkhb/cluster-candidate-5-0-rgjngf should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f68580>: 
    failed to create HCP cluster cluster-candidate-5-0-rgjngf: failed waiting for cluster="cluster-candidate-5-0-rgjngf" in resourcegroup="rg-candidate-5-0-9bqqftbhhkhb" to finish creating: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3baf4e11-4b0e-4700-b113-da3918d872d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3baf4e11-4b0e-4700-b113-da3918d872d5",
      "name": "3baf4e11-4b0e-4700-b113-da3918d872d5",
      "status": "Failed",
      "startTime": "2026-09-16T09:21:27.799687923Z",
      "endTime": "2026-09-16T09:50:32.506640388Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 18m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010be048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lq45ld6h888d" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a",
      "name": "76270801-127d-4225-97dc-06e57322bf5a",
      "status": "Failed",
      "startTime": "2026-09-16T09:37:08.667361714Z",
      "endTime": "2026-09-16T10:01:12.766789046Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss13j30utlk9cm4u8n21tm903korv38 still exists (deletion dispatched at 2026-09-16T09:37:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010be048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lq45ld6h888d" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a",
      "name": "76270801-127d-4225-97dc-06e57322bf5a",
      "status": "Failed",
      "startTime": "2026-09-16T09:37:08.667361714Z",
      "endTime": "2026-09-16T10:01:12.766789046Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss13j30utlk9cm4u8n21tm903korv38 still exists (deletion dispatched at 2026-09-16T09:37:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000c2c360>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rmqvg72srtqc" to finish creating: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88e478a1-228a-4fba-bfa8-f2447c215df2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88e478a1-228a-4fba-bfa8-f2447c215df2",
      "name": "88e478a1-228a-4fba-bfa8-f2447c215df2",
      "status": "Failed",
      "startTime": "2026-09-16T09:21:39.030244884Z",
      "endTime": "2026-09-16T09:50:42.360079347Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000c2c360>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rmqvg72srtqc" to finish creating: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88e478a1-228a-4fba-bfa8-f2447c215df2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88e478a1-228a-4fba-bfa8-f2447c215df2",
      "name": "88e478a1-228a-4fba-bfa8-f2447c215df2",
      "status": "Failed",
      "startTime": "2026-09-16T09:21:39.030244884Z",
      "endTime": "2026-09-16T09:50:42.360079347Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 18m27s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011720a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-4cfzlz" in resourcegroup="rg-zstream-upgrade-5-0-2bmmvg6nm4m6" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "name": "a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "status": "Failed",
      "startTime": "2026-09-16T09:52:39.028633437Z",
      "endTime": "2026-09-16T10:16:42.563449333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss136411ov65aeqqn1ut2t4iprrujv9 still exists (deletion dispatched at 2026-09-16T09:52:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011720a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-4cfzlz" in resourcegroup="rg-zstream-upgrade-5-0-2bmmvg6nm4m6" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "name": "a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "status": "Failed",
      "startTime": "2026-09-16T09:52:39.028633437Z",
      "endTime": "2026-09-16T10:16:42.563449333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss136411ov65aeqqn1ut2t4iprrujv9 still exists (deletion dispatched at 2026-09-16T09:52:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000bfe3a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-wgdjtrkdh4nq" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d757bcf9-a836-4d4b-93e9-18f5495774b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d757bcf9-a836-4d4b-93e9-18f5495774b1",
      "name": "d757bcf9-a836-4d4b-93e9-18f5495774b1",
      "status": "Failed",
      "startTime": "2026-09-16T09:47:47.203138976Z",
      "endTime": "2026-09-16T10:11:52.474449775Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000bfe3a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-wgdjtrkdh4nq" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d757bcf9-a836-4d4b-93e9-18f5495774b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d757bcf9-a836-4d4b-93e9-18f5495774b1",
      "name": "d757bcf9-a836-4d4b-93e9-18f5495774b1",
      "status": "Failed",
      "startTime": "2026-09-16T09:47:47.203138976Z",
      "endTime": "2026-09-16T10:11:52.474449775Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008820c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-hz87cjqncdvb" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3",
      "name": "8e715185-33d5-494b-976c-25b2ab261cc3",
      "status": "Failed",
      "startTime": "2026-09-16T09:35:58.536453378Z",
      "endTime": "2026-09-16T10:00:02.536562643Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1371bjtto4nf5m6o0pid4pjvle6qc still exists (deletion dispatched at 2026-09-16T09:35:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008820c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-hz87cjqncdvb" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3",
      "name": "8e715185-33d5-494b-976c-25b2ab261cc3",
      "status": "Failed",
      "startTime": "2026-09-16T09:35:58.536453378Z",
      "endTime": "2026-09-16T10:00:02.536562643Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1371bjtto4nf5m6o0pid4pjvle6qc still exists (deletion dispatched at 2026-09-16T09:35:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000706300>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-h546tchcqt6d" to finish creating: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bee6c146-3516-4eb7-bb5d-77be8886d51f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bee6c146-3516-4eb7-bb5d-77be8886d51f",
      "name": "bee6c146-3516-4eb7-bb5d-77be8886d51f",
      "status": "Failed",
      "startTime": "2026-09-16T09:23:24.300104342Z",
      "endTime": "2026-09-16T09:52:32.410749237Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000706300>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-h546tchcqt6d" to finish creating: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bee6c146-3516-4eb7-bb5d-77be8886d51f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bee6c146-3516-4eb7-bb5d-77be8886d51f",
      "name": "bee6c146-3516-4eb7-bb5d-77be8886d51f",
      "status": "Failed",
      "startTime": "2026-09-16T09:23:24.300104342Z",
      "endTime": "2026-09-16T09:52:32.410749237Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0018ba120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-5p7pgd" in resourcegroup="rg-cp-ystream-upgrade-4-22-bd9zhbrlv7hq" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83958f5a-6d08-4efa-bef6-3bc600801633
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83958f5a-6d08-4efa-bef6-3bc600801633",
      "name": "83958f5a-6d08-4efa-bef6-3bc600801633",
      "status": "Failed",
      "startTime": "2026-09-16T09:51:01.642991125Z",
      "endTime": "2026-09-16T10:15:02.48831324Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0018ba120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-5p7pgd" in resourcegroup="rg-cp-ystream-upgrade-4-22-bd9zhbrlv7hq" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83958f5a-6d08-4efa-bef6-3bc600801633
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83958f5a-6d08-4efa-bef6-3bc600801633",
      "name": "83958f5a-6d08-4efa-bef6-3bc600801633",
      "status": "Failed",
      "startTime": "2026-09-16T09:51:01.642991125Z",
      "endTime": "2026-09-16T10:15:02.48831324Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001512108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-mwjj2q" in resourcegroup="admin-credential-lifecycle-test-q4n7ckbgk7dt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4",
      "name": "5124f997-7212-409c-99ae-fbd89a51e2f4",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:00.208113506Z",
      "endTime": "2026-09-16T10:08:02.509863824Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12vvsmbd0thdp0f88kc41u6pvi7c9 still exists (deletion dispatched at 2026-09-16T09:44:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001512108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-mwjj2q" in resourcegroup="admin-credential-lifecycle-test-q4n7ckbgk7dt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4",
      "name": "5124f997-7212-409c-99ae-fbd89a51e2f4",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:00.208113506Z",
      "endTime": "2026-09-16T10:08:02.509863824Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12vvsmbd0thdp0f88kc41u6pvi7c9 still exists (deletion dispatched at 2026-09-16T09:44:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012f78>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-g9bvp2" in resourcegroup="rg-candidate-4-20-pmdbfsk6v6zd" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429",
      "name": "834c2939-b42e-48bd-b340-960d8826a429",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:50.424611242Z",
      "endTime": "2026-09-16T10:13:52.492147666Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012f78>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-g9bvp2" in resourcegroup="rg-candidate-4-20-pmdbfsk6v6zd" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429",
      "name": "834c2939-b42e-48bd-b340-960d8826a429",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:50.424611242Z",
      "endTime": "2026-09-16T10:13:52.492147666Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0028a4438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j5zkl24x6bwt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827",
      "name": "5ecb9f85-876c-467d-91c9-980ce11c9827",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:19.070350042Z",
      "endTime": "2026-09-16T10:13:22.487477443Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0028a4438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j5zkl24x6bwt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827",
      "name": "5ecb9f85-876c-467d-91c9-980ce11c9827",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:19.070350042Z",
      "endTime": "2026-09-16T10:13:22.487477443Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001580078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dfbjv854d6w5" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "name": "d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:31.257109777Z",
      "endTime": "2026-09-16T10:00:32.532474531Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1375769r7k6ubu137jlbqtq9j35it still exists (deletion dispatched at 2026-09-16T09:36:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001580078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dfbjv854d6w5" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "name": "d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:31.257109777Z",
      "endTime": "2026-09-16T10:00:32.532474531Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1375769r7k6ubu137jlbqtq9j35it still exists (deletion dispatched at 2026-09-16T09:36:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019ec138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gql2drdp7tjr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "name": "92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "status": "Failed",
      "startTime": "2026-09-16T09:34:43.828694597Z",
      "endTime": "2026-09-16T09:58:52.519202639Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1361u3o4ev8vfck5hk2a42dp30j3q still exists (deletion dispatched at 2026-09-16T09:34:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019ec138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gql2drdp7tjr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "name": "92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "status": "Failed",
      "startTime": "2026-09-16T09:34:43.828694597Z",
      "endTime": "2026-09-16T09:58:52.519202639Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1361u3o4ev8vfck5hk2a42dp30j3q still exists (deletion dispatched at 2026-09-16T09:34:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f92288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-djgr48qrs9vr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "name": "a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:21.892022588Z",
      "endTime": "2026-09-16T10:00:22.50023585Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12t13cvqgvb69duuok3m0c1if4g29 still exists (deletion dispatched at 2026-09-16T09:36:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f92288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-djgr48qrs9vr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "name": "a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:21.892022588Z",
      "endTime": "2026-09-16T10:00:22.50023585Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12t13cvqgvb69duuok3m0c1if4g29 still exists (deletion dispatched at 2026-09-16T09:36:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00153e0f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-9dwsgxpprw8c" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "name": "9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:48.885828252Z",
      "endTime": "2026-09-16T10:08:52.4506315Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00153e0f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-9dwsgxpprw8c" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "name": "9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:48.885828252Z",
      "endTime": "2026-09-16T10:08:52.4506315Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 20 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:35:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:35:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:35:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:35:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:35:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="ingress-operator", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u/ingress-operator has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:35:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="dns-operator", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u/dns-operator has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="ingress-operator", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u/ingress-operator has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1371bjtto4nf5m6o0pid4pjvle6qc-w3u2s1n8u7b7c4o/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12vvsmbd0thdp0f88kc41u6pvi7c9-y7r0c6f2g8v3k8g/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="dns-operator", endpoint="http", environment="ci01", instance="10.128.64.187:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u", pod="arohcp-monitor-kube-state-metrics-574f657d96-x7c5l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss12t13cvqgvb69duuok3m0c1if4g29-q1k5h2w4a4a1f5u/dns-operator has not matched the expected number of replicas for longer than 30 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:36:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:44:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:44:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss1361u3o4ev8vfck5hk2a42dp30j3q-basic-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:44:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:44:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5539456-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.154:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5539456/providers/microsoft.monitor/accounts/hcps-j5539456", namespace="ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b", pod="arohcp-monitor-kube-state-metrics-574f657d96-qmqb6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss13j2abs333rpsqenh1eeh8ha7bprr-e2y3l7s9r3u6x1b/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:37:46Z
  Ended:       2026-09-16T10:02:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5539456-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss143dkb8crn6be34gbib5ir298a46p-cilium-cluster", pod="router-d8b5b89dd-9h8gj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss143dkb8crn6be34gbib5ir298a46p-cilium-cluster/router-d8b5b89dd-9h8gj has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00045b6a0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-tbrwcb86jfr2" to finish creating: GET https://rp.j2607744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/100f602b-1abd-4d84-a8d9-1fc862b0b35f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/100f602b-1abd-4d84-a8d9-1fc862b0b35f",
      "name": "100f602b-1abd-4d84-a8d9-1fc862b0b35f",
      "status": "Failed",
      "startTime": "2026-09-16T08:49:15.881620207Z",
      "endTime": "2026-09-16T09:18:21.907934243Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00045b6a0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-tbrwcb86jfr2" to finish creating: GET https://rp.j2607744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/100f602b-1abd-4d84-a8d9-1fc862b0b35f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/100f602b-1abd-4d84-a8d9-1fc862b0b35f",
      "name": "100f602b-1abd-4d84-a8d9-1fc862b0b35f",
      "status": "Failed",
      "startTime": "2026-09-16T08:49:15.881620207Z",
      "endTime": "2026-09-16T09:18:21.907934243Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:09:50Z
  Ended:       2026-09-16T09:39:52Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2607744-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2607744/providers/microsoft.monitor/accounts/hcps-j2607744", namespace="ocm-arohcpci01-2ss0k398kmvs03u4ijn3q0ehd6e40uln-agg-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-nwfmp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss0k398kmvs03u4ijn3q0ehd6e40uln-agg-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--w2v2r67444v5/cluster-hs-pre-7fdw2l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e546a0>: 
    failed to create HCP cluster cluster-hs-pre-7fdw2l: failed waiting for cluster="cluster-hs-pre-7fdw2l" in resourcegroup="rg-hs-pre--w2v2r67444v5" to finish creating: GET https://rp.j9459200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bcb8510-6d64-40ac-8f34-a62f6104e1e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bcb8510-6d64-40ac-8f34-a62f6104e1e0",
      "name": "1bcb8510-6d64-40ac-8f34-a62f6104e1e0",
      "status": "Failed",
      "startTime": "2026-09-16T09:04:50.385784561Z",
      "endTime": "2026-09-16T09:33:55.262197127Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--w2v2r67444v5/cluster-hs-pre-7fdw2l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e546a0>: 
    failed to create HCP cluster cluster-hs-pre-7fdw2l: failed waiting for cluster="cluster-hs-pre-7fdw2l" in resourcegroup="rg-hs-pre--w2v2r67444v5" to finish creating: GET https://rp.j9459200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bcb8510-6d64-40ac-8f34-a62f6104e1e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1bcb8510-6d64-40ac-8f34-a62f6104e1e0",
      "name": "1bcb8510-6d64-40ac-8f34-a62f6104e1e0",
      "status": "Failed",
      "startTime": "2026-09-16T09:04:50.385784561Z",
      "endTime": "2026-09-16T09:33:55.262197127Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc001064760>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-fxln9tn9zv6z" to finish creating: GET https://rp.j9459200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e3f68377-a68e-4f20-b0b6-eb88dfb8b7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e3f68377-a68e-4f20-b0b6-eb88dfb8b7dc",
      "name": "e3f68377-a68e-4f20-b0b6-eb88dfb8b7dc",
      "status": "Failed",
      "startTime": "2026-09-16T09:06:01.673328236Z",
      "endTime": "2026-09-16T09:35:05.314564461Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc001064760>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-fxln9tn9zv6z" to finish creating: GET https://rp.j9459200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e3f68377-a68e-4f20-b0b6-eb88dfb8b7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e3f68377-a68e-4f20-b0b6-eb88dfb8b7dc",
      "name": "e3f68377-a68e-4f20-b0b6-eb88dfb8b7dc",
      "status": "Failed",
      "startTime": "2026-09-16T09:06:01.673328236Z",
      "endTime": "2026-09-16T09:35:05.314564461Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b10860>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-7npmvs46gpv5" to finish creating: GET https://rp.j9459200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e154370e-5d92-4a60-b860-409b26880120
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e154370e-5d92-4a60-b860-409b26880120",
      "name": "e154370e-5d92-4a60-b860-409b26880120",
      "status": "Failed",
      "startTime": "2026-09-16T09:05:11.363311377Z",
      "endTime": "2026-09-16T09:34:15.227803795Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b10860>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-7npmvs46gpv5" to finish creating: GET https://rp.j9459200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e154370e-5d92-4a60-b860-409b26880120
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e154370e-5d92-4a60-b860-409b26880120",
      "name": "e154370e-5d92-4a60-b860-409b26880120",
      "status": "Failed",
      "startTime": "2026-09-16T09:05:11.363311377Z",
      "endTime": "2026-09-16T09:34:15.227803795Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:36:29Z
  Ended:       2026-09-16T09:42:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9459200-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9459200-svc).

alert fired 6 time(s) (5 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:15:02Z
  Ended:       2026-09-16T10:05:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9459200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0r8p1f5df24425nn4ov1oc2b2s9qv-d2n3f2p5e8a4e8g", pod="router-67f446dfd-cc8xg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0r8p1f5df24425nn4ov1oc2b2s9qv-d2n3f2p5e8a4e8g/router-67f446dfd-cc8xg has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T09:16:00Z
  Ended:       2026-09-16T09:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9459200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0rcsdsi36qo49ntkanduupf2g1ep1-q6m6u0v2u8u0a3w", pod="router-6959bd5fcf-gghkx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0rcsdsi36qo49ntkanduupf2g1ep1-q6m6u0v2u8u0a3w/router-6959bd5fcf-gghkx has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T09:17:01Z
  Ended:       2026-09-16T09:42:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9459200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0rid7on74rka1kfmd8b56931isqa8-w8i3f2p7v0w9p4u", pod="router-66f89d764-wmnzq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0rid7on74rka1kfmd8b56931isqa8-w8i3f2p7v0w9p4u/router-66f89d764-wmnzq has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T09:17:01Z
  Ended:       2026-09-16T09:31:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9459200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0rfch8u2p7djo0cafmu1q2bpj2mq1-n2z8d3k4t3x1f7v", pod="router-66bbd6fb7d-bkxxd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0rfch8u2p7djo0cafmu1q2bpj2mq1-n2z8d3k4t3x1f7v/router-66bbd6fb7d-bkxxd has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-16T09:47:03Z
  Ended:       2026-09-16T09:53:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9459200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss14p1akq7nrsvh7882pljokv1oijq0-h1u2z7i1y5s1m9c", pod="openshift-apiserver-796668569c-jjbgg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss14p1akq7nrsvh7882pljokv1oijq0-h1u2z7i1y5s1m9c/openshift-apiserver-796668569c-jjbgg has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:29:21Z
  Ended:       2026-09-16T09:49:20Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j5155200-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155200/providers/microsoft.monitor/accounts/services-j5155200", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j5155200-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).

time=2026-09-16T08:43:27.720Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T08:43:28.143Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6406656-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6406656-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:510fcd49-73a6-4643-b104-d1da046c7a87 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#510fcd49-73a6-4643-b104-d1da046c7a87 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T08:43:28.143Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:43:28.283Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T08:43:28.434Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T08:43:28.434Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:43:28.493Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:53:31.202Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T08:53:31.203Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T08:54:31.204Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T08:54:31.661Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6406656-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6406656-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:510fcd49-73a6-4643-b104-d1da046c7a87 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#510fcd49-73a6-4643-b104-d1da046c7a87 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T08:54:31.661Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:31.919Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T08:54:32.413Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:32.413Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:32.686Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T08:54:33.497Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:33.498Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:33.902Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:34.404Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T08:54:34.757Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T08:54:34.953Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:35.182Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:35.544Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:36.068Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T08:54:36.068Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:36.176Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T09:04:37.727Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T09:04:37.727Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T09:05:37.733Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T09:05:38.373Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6406656-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6406656-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:510fcd49-73a6-4643-b104-d1da046c7a87 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#510fcd49-73a6-4643-b104-d1da046c7a87 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T09:05:38.373Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:38.676Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T09:05:39.023Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:39.023Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:39.181Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T09:05:40.018Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:40.018Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:40.264Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:40.639Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T09:05:41.058Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T09:05:41.352Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:41.683Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:42.063Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:42.276Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T09:05:42.276Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:42.359Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T09:13:27.645Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-16T09:13:27.646Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T08:59:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6406656-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.98:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6406656/providers/microsoft.monitor/accounts/services-j6406656", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-h8gf2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T08:59:42Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6406656-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.98:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6406656/providers/microsoft.monitor/accounts/services-j6406656", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-h8gf2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T08:49:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6406656-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-jpn9f", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-jpn9f has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T08:49:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6406656-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-cp72m", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-cp72m has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T08:49:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6406656-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-bcjjv", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-bcjjv has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T08:49:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6406656-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-z2nhx", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-z2nhx has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T09:00:34Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j6406656-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j6406656-svc has been unreachable for 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc0010522c0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-9vtzm4" in resourcegroup="rg-np-noedge-9vtzm4-knfsxvjkqt2p" to finish creating: GET https://rp.j3290112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae91cd5-981b-49a0-a69e-9b3ed247379a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae91cd5-981b-49a0-a69e-9b3ed247379a",
      "name": "1ae91cd5-981b-49a0-a69e-9b3ed247379a",
      "status": "Failed",
      "startTime": "2026-09-16T09:25:52.734930194Z",
      "endTime": "2026-09-16T09:26:31.102447106Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc0010522c0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-9vtzm4" in resourcegroup="rg-np-noedge-9vtzm4-knfsxvjkqt2p" to finish creating: GET https://rp.j3290112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae91cd5-981b-49a0-a69e-9b3ed247379a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ae91cd5-981b-49a0-a69e-9b3ed247379a",
      "name": "1ae91cd5-981b-49a0-a69e-9b3ed247379a",
      "status": "Failed",
      "startTime": "2026-09-16T09:25:52.734930194Z",
      "endTime": "2026-09-16T09:26:31.102447106Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T09:46:29Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j3290112-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3290112/providers/microsoft.monitor/accounts/services-j3290112", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j3290112-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:25:35Z
  Ended:       2026-09-16T09:57:36Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3290112-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3290112-svc).

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:39:29Z
  Ended:       2026-09-16T09:50:27Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP991h5m", cluster="ci01-j3290112-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3290112/providers/microsoft.monitor/accounts/services-j3290112", severity="3", short_window="5m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j3290112-svc is above 5s on both the 1h and 5m windows (severe miss of the p99 < 1s ARM sync SLO).
Firing 2:
  State:       Resolved
  Started:     2026-09-16T10:10:33Z
  Ended:       2026-09-16T10:17:29Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP991h5m", cluster="ci01-j3290112-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3290112/providers/microsoft.monitor/accounts/services-j3290112", severity="3", short_window="5m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j3290112-svc is above 5s on both the 1h and 5m windows (severe miss of the p99 < 1s ARM sync SLO).

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:15:02Z
  Ended:       2026-09-16T09:25:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9507712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0r1rv65slu402q5o4341pkordffhr-d7b4p7w0c7w3c0o", pod="router-6cb9c55d7c-4j7ns", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0r1rv65slu402q5o4341pkordffhr-d7b4p7w0c7w3c0o/router-6cb9c55d7c-4j7ns has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T09:17:03Z
  Ended:       2026-09-16T09:29:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9507712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0r6qa08bae603bidfs62a1vfvaur8-c3q5v4q2r7f5t2m", pod="router-8596496b74-m9vph", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0r6qa08bae603bidfs62a1vfvaur8-c3q5v4q2r7f5t2m/router-8596496b74-m9vph has been in a non-ready state for longer than 5 minutes.

time=2026-09-16T08:27:04.451Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output description="Step infra-output\n  Kind: ARM\n  Template: templates/svc-infra-lookup.bicep\n  Parameters: configurations/svc-infra-lookup.tmpl.bicepparam"
time=2026-09-16T08:27:04.849Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output
time=2026-09-16T08:27:05.446Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output deployment=eaaad5ae26368062d8da5183fba1f1cf527c2665c3e53af22a660fa55c8dc157 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7918592-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Feaaad5ae26368062d8da5183fba1f1cf527c2665c3e53af22a660fa55c8dc157
time=2026-09-16T08:27:35.588Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7918592-svc/providers/Microsoft.Resources/deployments/eaaad5ae26368062d8da5183fba1f1cf527c2665c3e53af22a660fa55c8dc157/operationStatuses/08584120596603628303\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"DeploymentActive\\\",\\r\\n    \\\"message\\\": \\\"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/svc-kv-xehipxbjxawki' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '9/16/2026 8:27:06 AM' with correlationId '032c9251-9fc6-4ff2-b8e3-30d15533261b', and will expire at '9/23/2026 8:27:06 AM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-16T09:03:26.822Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 description="Step kubelet-ds\n Kind: Helm\n"
time=2026-09-16T09:03:27.157Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 values="map[enabled:true kubeletParametersImage:mcr.microsoft.com/aks/command/runtime@sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23]"
time=2026-09-16T09:03:27.157Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-09-16T09:03:27.284Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 namespace=kubelet-ds
time=2026-09-16T09:03:27.475Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 dryRun=false
time=2026-09-16T09:03:27.475Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-09-16T09:03:27.530Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-09-16T09:08:28.583Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/18\ncontext deadline exceeded"
time=2026-09-16T09:08:28.583Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/18\ncontext deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-yt in resource group hcp-underlay-ci01-yt
Unexpected error:
    <*fmt.wrapError | 0xc001004ba0>: 
    failed to get workspace services-yt: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.Monitor/accounts/services-yt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-yt in resource group hcp-underlay-ci01-yt
Unexpected error:
    <*fmt.wrapError | 0xc001004ba0>: 
    failed to get workspace services-yt: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.Monitor/accounts/services-yt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc000c94e20>: 
    failed to get workspace hcps-yt: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.Monitor/accounts/hcps-yt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc000c94e20>: 
    failed to get workspace hcps-yt: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.Monitor/accounts/hcps-yt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000a540c0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000a540c0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...

alert fired 43 time(s) (31 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T08:46:36Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e1d6e28cgcj8srsgji99eu63mo4i-a2r9l9g4p0t9a9l", pod="kube-state-metrics-hcp-8654bffc85-ps27z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e1d6e28cgcj8srsgji99eu63mo4i-a2r9l9g4p0t9a9l/kube-state-metrics-hcp-8654bffc85-ps27z has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T08:46:36Z
  Ended:       2026-09-16T09:04:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0duf4s9stu0aprgf4ol4j3c99jk2f-j8v4h6s4u8u8p4l", pod="kube-state-metrics-hcp-8654bffc85-cwpmw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0duf4s9stu0aprgf4ol4j3c99jk2f-j8v4h6s4u8u8p4l/kube-state-metrics-hcp-8654bffc85-cwpmw has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T08:47:34Z
  Ended:       2026-09-16T09:06:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0dtmd7phkt983ei3vk9l5hqge338m-z6z2n3u3s4c1i5m", pod="kube-state-metrics-hcp-8654bffc85-tvn6p", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0dtmd7phkt983ei3vk9l5hqge338m-z6z2n3u3s4c1i5m/kube-state-metrics-hcp-8654bffc85-tvn6p has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-16T08:48:34Z
  Ended:       2026-09-16T09:06:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e9nto594s9bprvj5apj58rjljcjh-n0r0v2f3i9x3l1e", pod="kube-state-metrics-hcp-8654bffc85-q4qrz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e9nto594s9bprvj5apj58rjljcjh-n0r0v2f3i9x3l1e/kube-state-metrics-hcp-8654bffc85-q4qrz has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-16T08:48:34Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e5616jettcj0e7uig00q85cf1bug-q7s6x3r1w1e1g4u", pod="kube-state-metrics-hcp-8654bffc85-p8q8r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e5616jettcj0e7uig00q85cf1bug-q7s6x3r1w1e1g4u/kube-state-metrics-hcp-8654bffc85-p8q8r has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-16T08:48:34Z
  Ended:       2026-09-16T09:03:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0drohbr45u43inc6u449jd9sevca4-sre-hcp-cluster", pod="kube-state-metrics-hcp-8654bffc85-sclrb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0drohbr45u43inc6u449jd9sevca4-sre-hcp-cluster/kube-state-metrics-hcp-8654bffc85-sclrb has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-16T08:48:34Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0ds27bmfd0lp6hvok9h387svmutrh-m7o7e1m4q4g5a0q", pod="kube-state-metrics-hcp-8654bffc85-kw5xd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0ds27bmfd0lp6hvok9h387svmutrh-m7o7e1m4q4g5a0q/kube-state-metrics-hcp-8654bffc85-kw5xd has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-16T08:48:34Z
  Ended:       2026-09-16T09:26:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e5482l6g3gjjck8dmgk09unp5n62-w4i9v7z8p2f3l7h", pod="kube-state-metrics-hcp-8654bffc85-c2lnm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e5482l6g3gjjck8dmgk09unp5n62-w4i9v7z8p2f3l7h/kube-state-metrics-hcp-8654bffc85-c2lnm has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-16T08:49:36Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e0se573hope116jkdbf3q1bsdf8b-cilium-cl", pod="kube-state-metrics-hcp-8654bffc85-xvxb9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e0se573hope116jkdbf3q1bsdf8b-cilium-cl/kube-state-metrics-hcp-8654bffc85-xvxb9 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-16T08:49:36Z
  Ended:       2026-09-16T09:02:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0dulm91k97f02qma0s0l25hkbd2na-basic-cluster", pod="kube-state-metrics-hcp-8654bffc85-nc62p", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0dulm91k97f02qma0s0l25hkbd2na-basic-cluster/kube-state-metrics-hcp-8654bffc85-nc62p has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-16T08:49:36Z
  Ended:       2026-09-16T09:22:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e299ccde1v5r1t1qpok7ges0ddr0-c7r4k0l0o7y7f7g", pod="kube-state-metrics-hcp-8654bffc85-884m9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e299ccde1v5r1t1qpok7ges0ddr0-c7r4k0l0o7y7f7g/kube-state-metrics-hcp-8654bffc85-884m9 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-16T08:49:36Z
  Ended:       2026-09-16T09:23:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e2d6ioi48inqqd9othg3996mh958-k1z1b8a3k4u4i9i", pod="kube-state-metrics-hcp-8654bffc85-cm49w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e2d6ioi48inqqd9othg3996mh958-k1z1b8a3k4u4i9i/kube-state-metrics-hcp-8654bffc85-cm49w has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-16T08:50:35Z
  Ended:       2026-09-16T09:06:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0dscenj77pmhb82oqe4hmj9hquq7f-a4i8a2i9w1m2q8f", pod="kube-state-metrics-hcp-8654bffc85-8426r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0dscenj77pmhb82oqe4hmj9hquq7f-a4i8a2i9w1m2q8f/kube-state-metrics-hcp-8654bffc85-8426r has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-16T08:50:35Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e1lj58vfiilrpgklc8aev2a8vhm2-y3c2u9u6p8i8f1j", pod="kube-state-metrics-hcp-8654bffc85-q85rl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e1lj58vfiilrpgklc8aev2a8vhm2-y3c2u9u6p8i8f1j/kube-state-metrics-hcp-8654bffc85-q85rl has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-16T08:50:35Z
  Ended:       2026-09-16T09:21:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e2brb3h1abmdb9q6t7s3jhggu4pa-t3s9n8v8m7h8n4b", pod="kube-state-metrics-hcp-8654bffc85-kqjcf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e2brb3h1abmdb9q6t7s3jhggu4pa-t3s9n8v8m7h8n4b/kube-state-metrics-hcp-8654bffc85-kqjcf has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0dtrihd5vifelghabc0c05hniti06-w6n8j9u9k7n7b1g", pod="kube-state-metrics-hcp-8654bffc85-9njg2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0dtrihd5vifelghabc0c05hniti06-w6n8j9u9k7n7b1g/kube-state-metrics-hcp-8654bffc85-9njg2 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:09:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e57qf4hvm7piged8okdoteu9ep64-c3r3v6g0k6b2t0y", pod="kube-state-metrics-hcp-8654bffc85-t5t9l", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e57qf4hvm7piged8okdoteu9ep64-c3r3v6g0k6b2t0y/kube-state-metrics-hcp-8654bffc85-t5t9l has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0dtbtkug6eg8cvsj1ip575q4h6g8k-q6g6h8g9t3b7p2f", pod="kube-state-metrics-hcp-8654bffc85-786vd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0dtbtkug6eg8cvsj1ip575q4h6g8k-q6g6h8g9t3b7p2f/kube-state-metrics-hcp-8654bffc85-786vd has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:20:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0dsa50hk3db67ji91qm6ggb7bgd5g-r1x4c0m1g1y2p3q", pod="kube-state-metrics-hcp-8654bffc85-xj7w4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0dsa50hk3db67ji91qm6ggb7bgd5g-r1x4c0m1g1y2p3q/kube-state-metrics-hcp-8654bffc85-xj7w4 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:13:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0dto8jouf46i1bb19k20eh7vjukqa-ea-cluster", pod="kube-state-metrics-hcp-8654bffc85-cz9jw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0dto8jouf46i1bb19k20eh7vjukqa-ea-cluster/kube-state-metrics-hcp-8654bffc85-cz9jw has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:25:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e0gpe5eqkn8mba3h9lv428t445t4-s3v5h2p8x5u6m6o", pod="kube-state-metrics-hcp-8654bffc85-tv58x", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e0gpe5eqkn8mba3h9lv428t445t4-s3v5h2p8x5u6m6o/kube-state-metrics-hcp-8654bffc85-tv58x has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:13:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e3ndrc2bqqgvpe0os0ru0qu81d4p-f5r7a5r7u7s6n9a", pod="kube-state-metrics-hcp-8654bffc85-qzm78", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e3ndrc2bqqgvpe0os0ru0qu81d4p-f5r7a5r7u7s6n9a/kube-state-metrics-hcp-8654bffc85-qzm78 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:09:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e5qo1hiar598l8vimo23riu0p3og-ea-listp6l6j7", pod="kube-state-metrics-hcp-8654bffc85-ffpsk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e5qo1hiar598l8vimo23riu0p3og-ea-listp6l6j7/kube-state-metrics-hcp-8654bffc85-ffpsk has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:34:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e5e2tf5cdj027ehc6v6447aemoft-l6c5i1r6w9j9a1g", pod="kube-state-metrics-hcp-8654bffc85-g42k6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e5e2tf5cdj027ehc6v6447aemoft-l6c5i1r6w9j9a1g/kube-state-metrics-hcp-8654bffc85-g42k6 has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:11:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e1g1dpcqdb9vlq6fnutb7ef82jou-n9x3t3q0v5w7k4b", pod="kube-state-metrics-hcp-8654bffc85-ml4d9", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e1g1dpcqdb9vlq6fnutb7ef82jou-n9x3t3q0v5w7k4b/kube-state-metrics-hcp-8654bffc85-ml4d9 has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:30:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0eh4306o4cvqj7ckcj3sgvioip60o-n1l9j4i7v8l0i3d", pod="kube-state-metrics-hcp-8654bffc85-wf9nr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0eh4306o4cvqj7ckcj3sgvioip60o-n1l9j4i7v8l0i3d/kube-state-metrics-hcp-8654bffc85-wf9nr has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:06:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e9gihrbuhf99vlkgkmat5gh1gj57-x9h0l9t2j6n2x9h", pod="kube-state-metrics-hcp-8654bffc85-s4594", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e9gihrbuhf99vlkgkmat5gh1gj57-x9h0l9t2j6n2x9h/kube-state-metrics-hcp-8654bffc85-s4594 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:20:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e1ink1tm5udcqm74e4ndq08g2ria-g4t9b4q5g1f3v9e", pod="kube-state-metrics-hcp-8654bffc85-kdd4f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e1ink1tm5udcqm74e4ndq08g2ria-g4t9b4q5g1f3v9e/kube-state-metrics-hcp-8654bffc85-kdd4f has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:41:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e2i0dvn6r1kqn5krgm1e1odtoe99-f6s4u2c4f9l9e2a", pod="kube-state-metrics-hcp-8654bffc85-5pxfx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e2i0dvn6r1kqn5krgm1e1odtoe99-f6s4u2c4f9l9e2a/kube-state-metrics-hcp-8654bffc85-5pxfx has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:20:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0e5afk4l93i8hn8lrot867h8r4cm4-a3r0x6m0c1h1t2j", pod="kube-state-metrics-hcp-8654bffc85-f2577", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0e5afk4l93i8hn8lrot867h8r4cm4-a3r0x6m0c1h1t2j/kube-state-metrics-hcp-8654bffc85-f2577 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-16T08:52:35Z
  Ended:       2026-09-16T09:13:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9843328-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss0du7tfltom0c1644hm0cgcokcqo6b-n3k9i3w8r8w9h2o", pod="kube-state-metrics-hcp-8654bffc85-2w4nd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss0du7tfltom0c1644hm0cgcokcqo6b-n3k9i3w8r8w9h2o/kube-state-metrics-hcp-8654bffc85-2w4nd has been in a non-ready state for longer than 5 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:13:39Z
  Ended:       2026-09-16T09:19:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="kube-state-metrics-hcp", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9843328/providers/microsoft.monitor/accounts/hcps-j9843328", namespace="ocm-arohcpci01-2ss0e2d6ioi48inqqd9othg3996mh958-k1z1b8a3k4u4i9i", pod="arohcp-monitor-kube-state-metrics-574f657d96-jlv8v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss0e2d6ioi48inqqd9othg3996mh958-k1z1b8a3k4u4i9i/kube-state-metrics-hcp has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T09:13:39Z
  Ended:       2026-09-16T09:19:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="kube-state-metrics-hcp", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9843328/providers/microsoft.monitor/accounts/hcps-j9843328", namespace="ocm-arohcpci01-2ss0e299ccde1v5r1t1qpok7ges0ddr0-c7r4k0l0o7y7f7g", pod="arohcp-monitor-kube-state-metrics-574f657d96-jlv8v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss0e299ccde1v5r1t1qpok7ges0ddr0-c7r4k0l0o7y7f7g/kube-state-metrics-hcp has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-16T09:13:39Z
  Ended:       2026-09-16T09:27:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9843328-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="kube-state-metrics-hcp", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9843328/providers/microsoft.monitor/accounts/hcps-j9843328", namespace="ocm-arohcpci01-2ss0e5482l6g3gjjck8dmgk09unp5n62-w4i9v7z8p2f3l7h", pod="arohcp-monitor-kube-state-metrics-574f657d96-jlv8v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss0e5482l6g3gjjck8dmgk09unp5n62-w4i9v7z8p2f3l7h/kube-state-metrics-hcp has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T08:53:50Z
  Ended:       2026-09-16T09:06:52Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j9843328-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9843328/providers/microsoft.monitor/accounts/services-j9843328", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j9843328-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T08:58:25Z
  Ended:       2026-09-16T09:33:27Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j9789056-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9789056/providers/microsoft.monitor/accounts/services-j9789056", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j9789056-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T08:27:37Z
  Ended:       2026-09-16T08:40:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6082816-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ss03n4euqvgrodqiodqlsm6e44lqi3v-p9n3x2f1x8p9s9a", pod="router-7589fd99c7-22p68", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ss03n4euqvgrodqiodqlsm6e44lqi3v-p9n3x2f1x8p9s9a/router-7589fd99c7-22p68 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c6ad00>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-xpjdxspbvkfj" to finish creating: GET https://rp.j7238272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d55f5f9-f6e8-4345-b2b0-d96a01eab89e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d55f5f9-f6e8-4345-b2b0-d96a01eab89e",
      "name": "4d55f5f9-f6e8-4345-b2b0-d96a01eab89e",
      "status": "Failed",
      "startTime": "2026-09-16T07:56:30.963864533Z",
      "endTime": "2026-09-16T08:25:39.96844469Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c6ad00>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-xpjdxspbvkfj" to finish creating: GET https://rp.j7238272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d55f5f9-f6e8-4345-b2b0-d96a01eab89e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4d55f5f9-f6e8-4345-b2b0-d96a01eab89e",
      "name": "4d55f5f9-f6e8-4345-b2b0-d96a01eab89e",
      "status": "Failed",
      "startTime": "2026-09-16T07:56:30.963864533Z",
      "endTime": "2026-09-16T08:25:39.96844469Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m2s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010c81a0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-xbxmwmzgbdf6" to finish creating: GET https://rp.j7238272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b2412533-f3f2-47d5-9634-4c30b5400a83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b2412533-f3f2-47d5-9634-4c30b5400a83",
      "name": "b2412533-f3f2-47d5-9634-4c30b5400a83",
      "status": "Failed",
      "startTime": "2026-09-16T07:55:54.64211789Z",
      "endTime": "2026-09-16T08:24:59.944024206Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0010c81a0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-xbxmwmzgbdf6" to finish creating: GET https://rp.j7238272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b2412533-f3f2-47d5-9634-4c30b5400a83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b2412533-f3f2-47d5-9634-4c30b5400a83",
      "name": "b2412533-f3f2-47d5-9634-4c30b5400a83",
      "status": "Failed",
      "startTime": "2026-09-16T07:55:54.64211789Z",
      "endTime": "2026-09-16T08:24:59.944024206Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-16T07:35:17.577Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T07:35:18.216Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-16T07:35:19.015Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=d9196c0050ee21da58d027182daf82fc87c36acf0a933ff3ded4321f5c85c815 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4826624-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd9196c0050ee21da58d027182daf82fc87c36acf0a933ff3ded4321f5c85c815
time=2026-09-16T08:05:17.276Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T08:04:47Z
  Ended:       2026-09-16T08:25:46Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j0148480-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0148480/providers/microsoft.monitor/accounts/services-j0148480", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j0148480-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T08:41:46Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j0148480-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0148480/providers/microsoft.monitor/accounts/services-j0148480", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j0148480-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T07:44:18Z
  Ended:       2026-09-16T07:50:17Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0148480-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} has exceeded 250ms over the past 30 minutes (cluster ci01-j0148480-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T07:38:34Z
  Ended:       2026-09-16T07:45:33Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7547264-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7547264-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T07:52:13Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j7547264-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7547264/providers/microsoft.monitor/accounts/services-j7547264", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j7547264-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000df6cd8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-pfttlc9b26s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "name": "ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "status": "Failed",
      "startTime": "2026-09-16T22:57:32.075098968Z",
      "endTime": "2026-09-16T23:16:32.974740274Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000df6cd8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-pfttlc9b26s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "name": "ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "status": "Failed",
      "startTime": "2026-09-16T22:57:32.075098968Z",
      "endTime": "2026-09-16T23:16:32.974740274Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001d82db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-d2hkj9sg5shk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001d82db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-d2hkj9sg5shk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d85560>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jzkmnxzs9c52": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jzkmnxzs9c52/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d85560>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jzkmnxzs9c52": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jzkmnxzs9c52/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000fd73e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dqvpfm666hzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000fd73e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dqvpfm666hzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00133e180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-pgw6tmx6nx48/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00133e180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-pgw6tmx6nx48/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002c97050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8h5dwhv96zkm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002c97050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8h5dwhv96zkm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f3cd80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b7c4p2nvk6d6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d",
      "name": "0259344c-f395-4714-abc8-4abafaaf707d",
      "status": "Failed",
      "startTime": "2026-09-16T23:18:34.031156052Z",
      "endTime": "2026-09-16T23:37:37.814154784Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f3cd80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b7c4p2nvk6d6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d",
      "name": "0259344c-f395-4714-abc8-4abafaaf707d",
      "status": "Failed",
      "startTime": "2026-09-16T23:18:34.031156052Z",
      "endTime": "2026-09-16T23:37:37.814154784Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc00139eea0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-q5hhjjfw9zhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc00139eea0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-q5hhjjfw9zhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00152ab10>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00152ab10>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002a4e018>: 
    context deadline exceeded
    cluster operators not available: console (no Available condition), dns (Available=False, Progressing=True, Degraded=True, Upgradeable=True), image-registry (Available=False, Progressing=True, Degraded=True), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (no Available condition), openshift-samples (no Available condition), storage (Degraded=False, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002a4e018>: 
    context deadline exceeded
    cluster operators not available: console (no Available condition), dns (Available=False, Progressing=True, Degraded=True, Upgradeable=True), image-registry (Available=False, Progressing=True, Degraded=True), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (no Available condition), openshift-samples (no Available condition), storage (Degraded=False, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown)
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016e5a88>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016e5a88>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0027deed0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wck4dz5gg7vn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0027deed0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wck4dz5gg7vn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl"
Unexpected error:
    <*fmt.wrapError | 0xc002c1a920>: 
    failed to create NodePool np-pbnkzl: failed starting nodepool creation "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl" in resourcegroup="rg-nightly-4-20-6c9ns9nbgrzm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-6c9ns9nbgrzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-pbnkzl/nodePools/np-pbnkzl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl"
Unexpected error:
    <*fmt.wrapError | 0xc002c1a920>: 
    failed to create NodePool np-pbnkzl: failed starting nodepool creation "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl" in resourcegroup="rg-nightly-4-20-6c9ns9nbgrzm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-6c9ns9nbgrzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-pbnkzl/nodePools/np-pbnkzl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc000facdc0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-m58t8j" resourcegroup="rg-np-dg-minor-m58t8j-trxh29mwfdlb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-m58t8j-trxh29mwfdlb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-m58t8j/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc000facdc0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-m58t8j" resourcegroup="rg-np-dg-minor-m58t8j-trxh29mwfdlb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-m58t8j-trxh29mwfdlb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-m58t8j/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc001d4bc80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-56f2xp2w67w8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc001d4bc80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-56f2xp2w67w8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8"
Unexpected error:
    <*fmt.wrapError | 0xc00121ee40>: 
    failed to create NodePool np-krkxx8: failed waiting for nodepool="np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8" in resourcegroup="rg-nightly-4-23-7l9rdmzm7f2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "name": "4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "status": "Failed",
      "startTime": "2026-09-16T23:37:25.694607859Z",
      "endTime": "2026-09-16T23:56:27.737971571Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8"
Unexpected error:
    <*fmt.wrapError | 0xc00121ee40>: 
    failed to create NodePool np-krkxx8: failed waiting for nodepool="np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8" in resourcegroup="rg-nightly-4-23-7l9rdmzm7f2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "name": "4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "status": "Failed",
      "startTime": "2026-09-16T23:37:25.694607859Z",
      "endTime": "2026-09-16T23:56:27.737971571Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc0004d4640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-c8fd7f" in resourcegroup="rg-np-version-upgrade-c8fd7f-kdnr5znbw57r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c8fd7f-kdnr5znbw57r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c8fd7f/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc0004d4640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-c8fd7f" in resourcegroup="rg-np-version-upgrade-c8fd7f-kdnr5znbw57r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c8fd7f-kdnr5znbw57r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c8fd7f/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001682660>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-wjvk4n2mqnjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001682660>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-wjvk4n2mqnjj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc00063d580>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-w9gvcr" in resourcegroup="rg-np-skip-minor-w9gvcr-bshw78xg4hsg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-w9gvcr-bshw78xg4hsg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-w9gvcr/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc00063d580>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-w9gvcr" in resourcegroup="rg-np-skip-minor-w9gvcr-bshw78xg4hsg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-w9gvcr-bshw78xg4hsg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-w9gvcr/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000f2fd40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-g8sp2mm6ks67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000f2fd40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-g8sp2mm6ks67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010b5a80>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-2wnkdzqz6d9w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2wnkdzqz6d9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010b5a80>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-2wnkdzqz6d9w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2wnkdzqz6d9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0012fd950>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-57279mxs72dc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0012fd950>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-57279mxs72dc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred