time=2026-09-19T00:59:12.578Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=regional step=adminapi-alert-rule-role description="Step adminapi-alert-rule-role\n  Kind: ARM\n  Template: templates/adminapi-alert-processing-rule-role.bicep\n  Parameters: configurations/adminapi-alert-processing-rule-role.tmpl.bicepparam"
time=2026-09-19T00:59:12.962Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=regional step=adminapi-alert-rule-role
time=2026-09-19T00:59:13.303Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=regional step=adminapi-alert-rule-role err="failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/73540b7e4361f329d8cc7347416ba379091383d8ff0e1f7fb10b7cc1abb08e96\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplateDeployment\",\n    \"message\": \"The template deployment failed with error: 'Authorization failed for template resource '964da0aa-1cca-5874-beb3-3734babb6fde' of type 'Microsoft.Authorization/roleDefinitions'. The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id 'c209f8df-52ae-48fb-98ea-380f58b04652' does not have permission to perform action 'Microsoft.Authorization/roleDefinitions/write' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Authorization/roleDefinitions/964da0aa-1cca-5874-beb3-3734babb6fde'.'.\"\n  }\n}\n--------------------------------------------------------------------------------\n"

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:16:42Z
  Ended:       2026-09-19T01:52:41Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j8883328-mgmt-1", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="5m"
  Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T01:38:43Z
  Ended:       2026-09-19T01:47:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j8883328-mgmt-2", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="5m"
  Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:39:45Z
  Ended:       2026-09-19T01:51:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability6h30m", burn_rate_tier="medium", cluster="ci01-j8883328-mgmt-2", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="30m"
  Description: SWIFT router-capacity availability is consuming its error budget at the medium burn rate.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T01:41:44Z
  Ended:       2026-09-19T01:56:44Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability6h30m", burn_rate_tier="medium", cluster="ci01-j8883328-mgmt-1", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="30m"
  Description: SWIFT router-capacity availability is consuming its error budget at the medium burn rate.

time=2026-09-19T13:26:44.501Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 description="Step kubelet-ds\n Kind: Helm\n"
time=2026-09-19T13:26:44.949Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 values=map[enabled:true]
time=2026-09-19T13:26:44.949Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1
time=2026-09-19T13:26:45.277Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 namespace=kubelet-ds
time=2026-09-19T13:26:45.468Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 dryRun=false
time=2026-09-19T13:26:45.468Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1
time=2026-09-19T13:26:45.524Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1
time=2026-09-19T13:31:46.698Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 err="resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/15\ncontext deadline exceeded"
time=2026-09-19T13:31:46.698Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/15\ncontext deadline exceeded"

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T23:35:51Z
  Ended:       2026-09-19T23:41:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6722432-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6722432-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T21:05:45Z
  Ended:       2026-09-19T21:11:46Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0061568-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0061568-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T21:05:08Z
  Ended:       2026-09-19T21:11:08Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3425408-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3425408-svc).

alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T20:45:34Z
  Ended:       2026-09-19T21:12:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8800640-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sua7p1uu2oiunlri0cqe315nk9q9n8f-n1o6p7z8w6z4r2q", pod="router-777b99f8d7-8drf8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sua7p1uu2oiunlri0cqe315nk9q9n8f-n1o6p7z8w6z4r2q/router-777b99f8d7-8drf8 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T20:46:34Z
  Ended:       2026-09-19T21:11:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8800640-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sua7fdtmm97d9evql41ln5ik252m94i-private-kas", pod="router-685c668c54-ftrhk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sua7fdtmm97d9evql41ln5ik252m94i-private-kas/router-685c668c54-ftrhk has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-19T20:46:34Z
  Ended:       2026-09-19T21:12:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8800640-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sua7v07mnvun6l156puv4d57ktoj9ph-cilium-cluster", pod="router-9c9b7989-jmkt7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sua7v07mnvun6l156puv4d57ktoj9ph-cilium-cluster/router-9c9b7989-jmkt7 has been in a non-ready state for longer than 5 minutes.

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T18:54:13Z
  Ended:       2026-09-19T19:00:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2215680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su88o47irf75plc21jkh6milgqqj80l-s6c0t0c0k6r3x8q", pod="router-6b99f777f5-vr4nm", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su88o47irf75plc21jkh6milgqqj80l-s6c0t0c0k6r3x8q/router-6b99f777f5-vr4nm has been in a non-ready state for longer than 5 minutes.

alert fired 27 time(s) (15 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-node-tuning-operator-786575b7cc-rqwdt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-node-tuning-operator-786575b7cc-rqwdt has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-image-registry-operator-7d5c8cfb-2qs65", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-image-registry-operator-7d5c8cfb-2qs65 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="csi-snapshot-controller-operator-5bf7f97b88-vcs98", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/csi-snapshot-controller-operator-5bf7f97b88-vcs98 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82us6cpqcoa1ofkcab6a61g68g84r-private-ingress", pod="router-58d6c54947-vtg65", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82us6cpqcoa1ofkcab6a61g68g84r-private-ingress/router-58d6c54947-vtg65 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="openshift-controller-manager-666cf5fc85-56gwb", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/openshift-controller-manager-666cf5fc85-56gwb has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su8434ht5kskjt0q79ufdqc9r17debr-agg-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su8434ht5kskjt0q79ufdqc9r17debr-agg-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-policy-controller-65b5b47d99-l5lwt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-policy-controller-65b5b47d99-l5lwt has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="ingress-operator-ffb8cff7-lfknp", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/ingress-operator-ffb8cff7-lfknp has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="openshift-route-controller-manager-b76f84b4d-qq97g", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/openshift-route-controller-manager-b76f84b4d-qq97g has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-version-operator-85487bbffd-ldcp7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-version-operator-85487bbffd-ldcp7 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-network-operator-9498c74fd-sn2n9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-network-operator-9498c74fd-sn2n9 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:23Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="control-plane-pki-operator-9cf9f89cb-n5sjj", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/control-plane-pki-operator-9cf9f89cb-n5sjj has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="dns-operator-c98595c87-q5r9m", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/dns-operator-c98595c87-q5r9m has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-storage-operator-565c99cf9-2wfg2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-storage-operator-565c99cf9-2wfg2 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-19T18:21:24Z
  Ended:       2026-09-19T18:27:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="certified-operators-catalog-65bc785bb-lxzgs", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/certified-operators-catalog-65bc785bb-lxzgs has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-19T21:47:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j0061568-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2zqr9", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2zqr9 (namespace arobit) on cluster ci01-j0061568-svc has averaged 124.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-19T19:20:34Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1596032-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-zb5zr", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-zb5zr (namespace arobit) on cluster ci01-j1596032-svc has averaged 124.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-19T12:10:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9539072-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6qrsm", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6qrsm (namespace arobit) on cluster ci01-j9539072-svc has averaged 124% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T23:51:12Z
  Ended:       2026-09-20T00:04:13Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j1465216-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j1465216-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T20:58:11Z
  Ended:       2026-09-19T21:04:12Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j8848768-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j8848768-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T11:44:47Z
  Ended:       2026-09-19T12:09:48Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9539072-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9539072-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000a06120>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wh7bztj94wh4" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "name": "a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:43.428610597Z",
      "endTime": "2026-09-19T21:03:49.814289269Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000a06120>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wh7bztj94wh4" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "name": "a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:43.428610597Z",
      "endTime": "2026-09-19T21:03:49.814289269Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d24520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fgvlnn5kdj25" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "name": "8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "status": "Failed",
      "startTime": "2026-09-19T20:35:08.292814934Z",
      "endTime": "2026-09-19T21:04:09.799274903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d24520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fgvlnn5kdj25" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "name": "8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "status": "Failed",
      "startTime": "2026-09-19T20:35:08.292814934Z",
      "endTime": "2026-09-19T21:04:09.799274903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0001da320>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rpq79wd7j7md" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa",
      "name": "6e591bf2-2064-49c7-857a-322459bd75aa",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:05.40086812Z",
      "endTime": "2026-09-19T21:03:09.741249314Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m39s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0001da320>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rpq79wd7j7md" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa",
      "name": "6e591bf2-2064-49c7-857a-322459bd75aa",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:05.40086812Z",
      "endTime": "2026-09-19T21:03:09.741249314Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m39s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T04:03:08Z
  Ended:       2026-09-19T04:21:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1862272-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.64.253:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862272/providers/microsoft.monitor/accounts/hcps-j1862272", namespace="ocm-arohcpci01-2str5vc78gg1rc75eamfsefvm7hjps76-p8y4e2f9o6p1q5z", pod="arohcp-monitor-kube-state-metrics-7955558448-8g9rn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2str5vc78gg1rc75eamfsefvm7hjps76-p8y4e2f9o6p1q5z/router has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:43:47Z
  Ended:       2026-09-19T01:58:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-6486647766-4mw7w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00094e420>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dk96slfsvrkf" to finish creating: GET https://rp.j8467840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "name": "3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "status": "Failed",
      "startTime": "2026-09-19T01:06:55.31906784Z",
      "endTime": "2026-09-19T01:36:02.073820212Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00094e420>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dk96slfsvrkf" to finish creating: GET https://rp.j8467840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "name": "3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "status": "Failed",
      "startTime": "2026-09-19T01:06:55.31906784Z",
      "endTime": "2026-09-19T01:36:02.073820212Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000cd7540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-zrpb7z" in resourcegroup="rg-np-dg-minor-zrpb7z-2p25vj8zqmn4" to finish creating: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "name": "bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "status": "Failed",
      "startTime": "2026-09-19T13:03:18.457829152Z",
      "endTime": "2026-09-19T13:22:23.311283757Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000cd7540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-zrpb7z" in resourcegroup="rg-np-dg-minor-zrpb7z-2p25vj8zqmn4" to finish creating: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "name": "bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "status": "Failed",
      "startTime": "2026-09-19T13:03:18.457829152Z",
      "endTime": "2026-09-19T13:22:23.311283757Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:28:47Z
  Ended:       2026-09-19T01:54:50Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-6486647766-4mw7w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T01:29:59Z
  Ended:       2026-09-19T01:56:46Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-6486647766-4mw7w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:30:45Z
  Ended:       2026-09-19T01:58:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="daf93e4d-d43f-49cd-bbc8-4620650d35fc"
  Description: Pod ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T11:30:53Z
  Ended:       2026-09-19T11:44:53Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4648192-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fce138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xvkwtqlz7wnc" to finish deleting: GET https://rp.j8827264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "name": "7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "status": "Failed",
      "startTime": "2026-09-19T14:20:22.933309802Z",
      "endTime": "2026-09-19T14:44:28.067256394Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2su42fk9ipo2ddicrn6ki899u5dtl5la still exists (deletion dispatched at 2026-09-19T14:20:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fce138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xvkwtqlz7wnc" to finish deleting: GET https://rp.j8827264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "name": "7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "status": "Failed",
      "startTime": "2026-09-19T14:20:22.933309802Z",
      "endTime": "2026-09-19T14:44:28.067256394Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2su42fk9ipo2ddicrn6ki899u5dtl5la still exists (deletion dispatched at 2026-09-19T14:20:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d2ad80>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed waiting for cluster="np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt" to finish creating: GET https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: Canceled
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "name": "ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "status": "Canceled",
      "startTime": "2026-09-19T18:21:32.692382703Z",
      "endTime": "2026-09-19T18:21:36.063277066Z",
      "error": {
        "code": "Canceled",
        "message": "This operation was superseded by another"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d2ad80>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed waiting for cluster="np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt" to finish creating: GET https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: Canceled
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "name": "ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "status": "Canceled",
      "startTime": "2026-09-19T18:21:32.692382703Z",
      "endTime": "2026-09-19T18:21:36.063277066Z",
      "error": {
        "code": "Canceled",
        "message": "This operation was superseded by another"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00167c880>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed starting cluster creation "np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt": PUT https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00167c880>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed starting cluster creation "np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt": PUT https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-19T19:26:02.513Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1 description="Step infra\n  Kind: ARM\n  Template: templates/mgmt-infra.bicep\n  Parameters: configurations/mgmt-infra.tmpl.bicepparam"
time=2026-09-19T19:26:03.032Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1
time=2026-09-19T19:26:04.005Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1 deployment=6b08ae6c0b84b662e34009b99333555395423c7dc45a028e443e73278d202759 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3811200-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6b08ae6c0b84b662e34009b99333555395423c7dc45a028e443e73278d202759
time=2026-09-19T19:28:34.855Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3811200-mgmt-1/providers/Microsoft.Resources/deployments/6b08ae6c0b84b662e34009b99333555395423c7dc45a028e443e73278d202759/operationStatuses/08584117609220848049\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3811200-mgmt-1/providers/Microsoft.Resources/deployments/ah-ci01-me-j3811200-1\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-me-j3811200-1' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001364120>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-49gn4qjgmx5r" to finish creating: GET https://rp.j7481216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110",
      "name": "5d405444-e5c6-413b-8e97-5fb154fce110",
      "status": "Failed",
      "startTime": "2026-09-19T11:23:33.676780621Z",
      "endTime": "2026-09-19T11:42:34.970958081Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001364120>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-49gn4qjgmx5r" to finish creating: GET https://rp.j7481216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110",
      "name": "5d405444-e5c6-413b-8e97-5fb154fce110",
      "status": "Failed",
      "startTime": "2026-09-19T11:23:33.676780621Z",
      "endTime": "2026-09-19T11:42:34.970958081Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc00297dbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-fsdprlzmvbsr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc00297dbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-fsdprlzmvbsr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00175eea0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cj457ljpp84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00175eea0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cj457ljpp84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015bd1a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8c9pqlg82rcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015bd1a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8c9pqlg82rcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0013a9700>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-t2vqgb" resourcegroup="rg-np-dg-minor-t2vqgb-v48wcn4f6v4f": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-t2vqgb-v48wcn4f6v4f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-t2vqgb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0013a9700>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-t2vqgb" resourcegroup="rg-np-dg-minor-t2vqgb-v48wcn4f6v4f": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-t2vqgb-v48wcn4f6v4f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-t2vqgb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00093ba00>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-w54s7z" in resourcegroup="rg-np-skip-minor-w54s7z-sj4xc78f5vzx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-w54s7z-sj4xc78f5vzx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-w54s7z/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00093ba00>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-w54s7z" in resourcegroup="rg-np-skip-minor-w54s7z-sj4xc78f5vzx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-w54s7z-sj4xc78f5vzx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-w54s7z/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p"
Unexpected error:
    <*fmt.wrapError | 0xc001386c00>: 
    failed to create NodePool np-8gnl2p: failed starting nodepool creation "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p" in resourcegroup="rg-nightly-4-20-pwndrzvtlxs6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-pwndrzvtlxs6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-8gnl2p/nodePools/np-8gnl2p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p"
Unexpected error:
    <*fmt.wrapError | 0xc001386c00>: 
    failed to create NodePool np-8gnl2p: failed starting nodepool creation "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p" in resourcegroup="rg-nightly-4-20-pwndrzvtlxs6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-pwndrzvtlxs6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-8gnl2p/nodePools/np-8gnl2p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000a35460>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-zs4kjp29q62s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "name": "d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "status": "Failed",
      "startTime": "2026-09-19T23:48:22.46928069Z",
      "endTime": "2026-09-20T00:07:28.436280978Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000a35460>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-zs4kjp29q62s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "name": "d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "status": "Failed",
      "startTime": "2026-09-19T23:48:22.46928069Z",
      "endTime": "2026-09-20T00:07:28.436280978Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00106b280>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-mbf8fbn585g6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-mbf8fbn585g6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00106b280>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-mbf8fbn585g6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-mbf8fbn585g6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ac15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ac15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00160b7d0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rzlnqd88ksbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "name": "0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "status": "Failed",
      "startTime": "2026-09-19T02:54:29.386299179Z",
      "endTime": "2026-09-19T03:13:30.082275878Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-sgbzk-2tmm2": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-sgbzk-2tmm2" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rzlnqd88ksbr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sgbzk-2tmm2/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-sgbzk-2tmm2' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00160b7d0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rzlnqd88ksbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "name": "0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "status": "Failed",
      "startTime": "2026-09-19T02:54:29.386299179Z",
      "endTime": "2026-09-19T03:13:30.082275878Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-sgbzk-2tmm2": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-sgbzk-2tmm2" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rzlnqd88ksbr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sgbzk-2tmm2/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-sgbzk-2tmm2' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00381c7e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-8j5t9t6fn8sk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71",
      "name": "d87907d8-e7ff-4012-ba96-a28fad371f71",
      "status": "Failed",
      "startTime": "2026-09-19T03:13:36.546108406Z",
      "endTime": "2026-09-19T03:32:43.229867603Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00381c7e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-8j5t9t6fn8sk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71",
      "name": "d87907d8-e7ff-4012-ba96-a28fad371f71",
      "status": "Failed",
      "startTime": "2026-09-19T03:13:36.546108406Z",
      "endTime": "2026-09-19T03:32:43.229867603Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fa6100>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-jrsfr4h4jx7t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jrsfr4h4jx7t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fa6100>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-jrsfr4h4jx7t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jrsfr4h4jx7t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000ac24e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-977lq5c64l4r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-977lq5c64l4r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000ac24e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-977lq5c64l4r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-977lq5c64l4r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0009a4540>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-qqzhhqdh5xkq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-qqzhhqdh5xkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0009a4540>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-qqzhhqdh5xkq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-qqzhhqdh5xkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred