fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:250]: failed to verify cluster operators on cluster "cluster-candidate-4-23-df9q9l"
Unexpected error:
    <*errors.joinError | 0x124e8626e318>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb": nodes "r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x124e8515f040>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\": nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x124e8515f000>{
                    msg: "failed to label node r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\": nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x124e8515efc0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\": nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x124e8530e280>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:250]: failed to verify cluster operators on cluster "cluster-candidate-4-23-df9q9l"
Unexpected error:
    <*errors.joinError | 0x124e8626e318>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb": nodes "r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x124e8515f040>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\": nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x124e8515f000>{
                    msg: "failed to label node r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\": nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x124e8515efc0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\": nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x124e8530e280>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "r2v6x5y1r4f5h5a-np-df9q9l-fvq4n-7rzfb",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
    <*fmt.wrapError | 0x11989c84120>: 
    DNS for route host agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x11989c840a0>{
            msg: "DNS for agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
    <*fmt.wrapError | 0x11989c84120>: 
    DNS for route host agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x11989c840a0>{
            msg: "DNS for agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-kb9js.apps.aro.ingress-cert-le.a11n.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-zdl7h9"
Unexpected error:
    <*fmt.wrapError | 0x7f0cb444140>: 
    DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x7f0cb818340>{
            msg: "DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-zdl7h9"
Unexpected error:
    <*fmt.wrapError | 0x7f0cb444140>: 
    DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x7f0cb818340>{
            msg: "DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

time=2026-10-24T16:56:57.345Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T16:56:58.695Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T16:56:58.695Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T16:56:59.317Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T16:56:59.317Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T16:56:59.317Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j5468416-svc
time=2026-10-24T17:12:01.792Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"

alert fired 26 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-pbtkv", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-pbtkv has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-82j8b", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-82j8b has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-f27gs", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-f27gs has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-lwgdc", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-lwgdc has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-vs5np", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-vs5np has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-87dfl", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-87dfl has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-qpl82", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-qpl82 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-5q9wz", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-5q9wz has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-q7t7r", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-q7t7r has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-7llr5", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-7llr5 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-lxj96", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-lxj96 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-mcdmv", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-mcdmv has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-swlsf", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-swlsf has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-tkgxq", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-tkgxq has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-92mtz", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-92mtz has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-7vvmn", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-7vvmn has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-tb8wn", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-tb8wn has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-cv95p", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-cv95p has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-qpvlz", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-qpvlz has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-zfl7b", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-zfl7b has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-4wdk9", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-4wdk9 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-9hkcm", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-9hkcm has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-699hx", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-699hx has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-hdfbt", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-hdfbt has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:02:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-cxrkz", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-cxrkz has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-10-24T17:03:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-4hnj4", region="centralus", severity="warning"
  Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-4hnj4 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x2617933fd6e0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x2617933fd6e0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-2v886t"
Expected success, but got an error:
    <*fmt.wrapError | 0x2ea218b7e820>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2ea218b7e780>{
            observation: <*errors.errorString | 0x2ea218d2a9b0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2ea218b7e760>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-2v886t"
Expected success, but got an error:
    <*fmt.wrapError | 0x2ea218b7e820>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2ea218b7e780>{
            observation: <*errors.errorString | 0x2ea218d2a9b0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2ea218b7e760>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x11da4f3f5de0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x11da4f3f5de0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2tqhb4"
Expected success, but got an error:
    <*fmt.wrapError | 0x2f8a3019d120>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2f8a3019d0e0>{
            observation: <*errors.errorString | 0x2f8a325804f0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2f8a3019cf00>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2tqhb4"
Expected success, but got an error:
    <*fmt.wrapError | 0x2f8a3019d120>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2f8a3019d0e0>{
            observation: <*errors.errorString | 0x2f8a325804f0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2f8a3019cf00>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x13fb9bc14400>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x13fb9bc14400>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

time=2026-10-24T12:42:08.055Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T12:42:09.448Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T12:42:09.448Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T12:42:10.162Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T12:42:10.162Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T12:42:10.162Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j1222656-svc
time=2026-10-24T12:57:12.942Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x232a21da8bc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5cnqwcb679kx" to finish creating: GET https://rp.j7326208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eee303a2-7618-4a6c-a91a-e282ee1845c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eee303a2-7618-4a6c-a91a-e282ee1845c4",
      "name": "eee303a2-7618-4a6c-a91a-e282ee1845c4",
      "status": "Failed",
      "startTime": "2026-10-24T11:30:35.210037853Z",
      "endTime": "2026-10-24T11:49:35.657573237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y3p5h5t6y6r0a2w-np-1-656n9-dj76m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5cnqwcb679kx--managed/providers/Microsoft.Compute/virtualMachines/y3p5h5t6y6r0a2w-np-1-656n9-dj76m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y3p5h5t6y6r0a2w-np-1-656n9-dj76m: Failed: interfaces failed to create or update. err: failed to get existing resource autoscaling-cluster-5cnqwcb679kx--managed/y3p5h5t6y6r0a2w-np-1-656n9-dj76m-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5cnqwcb679kx--managed/providers/Microsoft.Network/networkInterfaces/y3p5h5t6y6r0a2w-np-1-656n9-dj76m-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '07aa0e83-353e-444c-9f1d-d023ac3c5396' exceeded. Please try again after '3' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7326208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eee303a2-7618-4a6c-a91a-e282ee1845c4
            correlation ID: 1204c613-1fa2-4780-b8d4-cf6423a998a8
            client request ID: 70ebbcc0-cca4-424d-8160-3551b5e8f506
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x232a21da8bc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5cnqwcb679kx" to finish creating: GET https://rp.j7326208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eee303a2-7618-4a6c-a91a-e282ee1845c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eee303a2-7618-4a6c-a91a-e282ee1845c4",
      "name": "eee303a2-7618-4a6c-a91a-e282ee1845c4",
      "status": "Failed",
      "startTime": "2026-10-24T11:30:35.210037853Z",
      "endTime": "2026-10-24T11:49:35.657573237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y3p5h5t6y6r0a2w-np-1-656n9-dj76m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5cnqwcb679kx--managed/providers/Microsoft.Compute/virtualMachines/y3p5h5t6y6r0a2w-np-1-656n9-dj76m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y3p5h5t6y6r0a2w-np-1-656n9-dj76m: Failed: interfaces failed to create or update. err: failed to get existing resource autoscaling-cluster-5cnqwcb679kx--managed/y3p5h5t6y6r0a2w-np-1-656n9-dj76m-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-5cnqwcb679kx--managed/providers/Microsoft.Network/networkInterfaces/y3p5h5t6y6r0a2w-np-1-656n9-dj76m-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '07aa0e83-353e-444c-9f1d-d023ac3c5396' exceeded. Please try again after '3' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7326208.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eee303a2-7618-4a6c-a91a-e282ee1845c4
            correlation ID: 1204c613-1fa2-4780-b8d4-cf6423a998a8
            client request ID: 70ebbcc0-cca4-424d-8160-3551b5e8f506
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-9vhc67"
Expected success, but got an error:
    <*fmt.wrapError | 0x2f35f4c5c1a0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2f35f4c5c120>{
            observation: <*errors.errorString | 0x2f35f6f1e830>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2f35f4c5c0c0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-9vhc67"
Expected success, but got an error:
    <*fmt.wrapError | 0x2f35f4c5c1a0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2f35f4c5c120>{
            observation: <*errors.errorString | 0x2f35f6f1e830>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2f35f4c5c0c0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x191d86000da0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x191d86000da0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-2ll92l"
Expected success, but got an error:
    <*fmt.wrapError | 0xd8e10b4e180>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0xd8e11f02690>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-2ll92l"
Expected success, but got an error:
    <*fmt.wrapError | 0xd8e10b4e180>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0xd8e11f02690>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:250]: failed to verify cluster operators on cluster "cluster-candidate-5-1-r4gwvg"
Unexpected error:
    <*errors.joinError | 0x32cae7fe4300>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h": nodes "l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x32cae7d8f2c0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\": nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x32cae7d8f2a0>{
                    msg: "failed to label node l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\": nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x32cae7d8f280>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\": nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x32cae7a38dc0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:250]: failed to verify cluster operators on cluster "cluster-candidate-5-1-r4gwvg"
Unexpected error:
    <*errors.joinError | 0x32cae7fe4300>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h": nodes "l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x32cae7d8f2c0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\": nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x32cae7d8f2a0>{
                    msg: "failed to label node l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\": nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x32cae7d8f280>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\": nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x32cae7a38dc0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "l8x3u9g8g1l7q7r-np-r4gwvg-vbqhr-7vz2h",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x274951b76fe0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x274951b76fe0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

alert fired 6 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-24T10:00:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-24T10:00:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-24T10:09:36Z
  Ended:       2026-10-24T10:18:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-24T10:09:36Z
  Ended:       2026-10-24T10:18:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-24T10:10:38Z
  Ended:       2026-10-24T10:21:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-24T10:10:38Z
  Ended:       2026-10-24T10:21:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-24T10:10:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-24T10:10:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-24T09:35:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-58f4f6fb5b-wj69n", region="centralus", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-58f4f6fb5b-wj69n has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc346b690c20>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc346b690c20>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 601.097s.
Expected success, but got an error:
    <*errors.errorString | 0x11218fd9090>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11200, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11200, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 601.097s.
Expected success, but got an error:
    <*errors.errorString | 0x11218fd9090>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11200, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11200, acm-agent=0",
    }

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-24T05:47:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1245952-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.161:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1245952/providers/microsoft.monitor/accounts/services-j1245952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-s9thv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ff2d3a94-ed8f-48bd-b318-c937cce68e20"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-s9thv (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-24T05:47:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1245952-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.161:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1245952/providers/microsoft.monitor/accounts/services-j1245952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-s9thv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ff2d3a94-ed8f-48bd-b318-c937cce68e20"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-s9thv (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-24T05:49:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1245952-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.30:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1245952/providers/microsoft.monitor/accounts/services-j1245952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-lpxww", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="346ba3f6-80dd-4a56-add6-18f9140c124e"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-lpxww (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-24T05:51:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1245952-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.30:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1245952/providers/microsoft.monitor/accounts/services-j1245952", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-lpxww", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="346ba3f6-80dd-4a56-add6-18f9140c124e"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-lpxww (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x286c46f305a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j4319616-mgmt-2/ocm-arohcpci01-2tkvarldubqmpnbonti2lc7tvjkd8h7v-m2m5f8i0z1w3d9n: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j4319616-mgmt-2/ocm-arohcpci01-2tkvarldubqmpnbonti2lc7tvjkd8h7v-m2m5f8i0z1w3d9n: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        err: <*verifiers.pollObservationError | 0x286c46f30580>{
            observation: <*errors.errorString | 0x286c458cdbb0>{
                s: "dev/canadacentral/ci01-j4319616-mgmt-2/ocm-arohcpci01-2tkvarldubqmpnbonti2lc7tvjkd8h7v-m2m5f8i0z1w3d9n: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
            },
            interruption: <*fmt.wrapError | 0x286c46f30560>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T05:33:22.4488158Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x286c4628c1e0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x286c46f30540>{
                        msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T05:33:22.4488158Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x286c478a50e0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x286c46f305a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j4319616-mgmt-2/ocm-arohcpci01-2tkvarldubqmpnbonti2lc7tvjkd8h7v-m2m5f8i0z1w3d9n: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j4319616-mgmt-2/ocm-arohcpci01-2tkvarldubqmpnbonti2lc7tvjkd8h7v-m2m5f8i0z1w3d9n: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        err: <*verifiers.pollObservationError | 0x286c46f30580>{
            observation: <*errors.errorString | 0x286c458cdbb0>{
                s: "dev/canadacentral/ci01-j4319616-mgmt-2/ocm-arohcpci01-2tkvarldubqmpnbonti2lc7tvjkd8h7v-m2m5f8i0z1w3d9n: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
            },
            interruption: <*fmt.wrapError | 0x286c46f30560>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T05:33:22.4488158Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x286c4628c1e0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x286c46f30540>{
                        msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wkh9kwhf6mgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T05:33:22.4488158Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x286c478a50e0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x1f6cce913900>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b559ft" in resourcegroup="rg-np-version-upgrade-b559ft-j4qdwfs6szxx" to finish creating: GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e",
      "name": "489dec27-e4f4-49eb-a330-6052b204ad8e",
      "status": "Failed",
      "startTime": "2026-10-24T05:53:26.952860579Z",
      "endTime": "2026-10-24T06:12:28.455904126Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
            correlation ID: d5af14b9-8e48-4974-af98-abb8c215a6d4
            client request ID: 1283d9b2-c265-4f0a-97ea-52c82d8cd670
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x1f6cce913900>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b559ft" in resourcegroup="rg-np-version-upgrade-b559ft-j4qdwfs6szxx" to finish creating: GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e",
      "name": "489dec27-e4f4-49eb-a330-6052b204ad8e",
      "status": "Failed",
      "startTime": "2026-10-24T05:53:26.952860579Z",
      "endTime": "2026-10-24T06:12:28.455904126Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
            correlation ID: d5af14b9-8e48-4974-af98-abb8c215a6d4
            client request ID: 1283d9b2-c265-4f0a-97ea-52c82d8cd670
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x3d4f883a52d0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x3d4f883a52d0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-8db7r8"
Expected success, but got an error:
    <*fmt.wrapError | 0x31b88a1146c0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x31b88a1146a0>{
            observation: <*errors.errorString | 0x31b88a000c30>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x31b88a114640>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-8db7r8"
Expected success, but got an error:
    <*fmt.wrapError | 0x31b88a1146c0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x31b88a1146a0>{
            observation: <*errors.errorString | 0x31b88a000c30>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x31b88a114640>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-84tw8j"
Expected success, but got an error:
    <*fmt.wrapError | 0x347780605600>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x3477806055c0>{
            observation: <*errors.errorString | 0x34778111e160>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x3477806055a0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-84tw8j"
Expected success, but got an error:
    <*fmt.wrapError | 0x347780605600>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x3477806055c0>{
            observation: <*errors.errorString | 0x34778111e160>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x3477806055a0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x9365e5f86a0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x9365e5f86a0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x298d72d44040>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-gw598c" in resourcegroup="rg-np-noedge-gw598c-2zsqpwzqbmb8" to finish creating: GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
      "name": "ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
      "status": "Failed",
      "startTime": "2026-10-24T05:54:16.071407678Z",
      "endTime": "2026-10-24T06:13:22.793053391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
            correlation ID: 149b615d-592f-486c-b354-94d297e4fa6d
            client request ID: eb91eb91-fd3e-493e-b389-b15c2dca94ef
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
    <*fmt.wrapError | 0x298d72d44040>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-gw598c" in resourcegroup="rg-np-noedge-gw598c-2zsqpwzqbmb8" to finish creating: GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
      "name": "ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
      "status": "Failed",
      "startTime": "2026-10-24T05:54:16.071407678Z",
      "endTime": "2026-10-24T06:13:22.793053391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
            correlation ID: 149b615d-592f-486c-b354-94d297e4fa6d
            client request ID: eb91eb91-fd3e-493e-b389-b15c2dca94ef
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.002s.
cert-manager deployment should complete rollout with workload identity
Expected success, but got an error:
    <*errors.errorString | 0x224df5000400>: 
    cert-manager rollout in progress: updated=1 ready=1 desired=2
    {
        s: "cert-manager rollout in progress: updated=1 ready=1 desired=2",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.002s.
cert-manager deployment should complete rollout with workload identity
Expected success, but got an error:
    <*errors.errorString | 0x224df5000400>: 
    cert-manager rollout in progress: updated=1 ready=1 desired=2
    {
        s: "cert-manager rollout in progress: updated=1 ready=1 desired=2",
    }

time=2026-10-24T03:19:41.940Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T03:19:44.124Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T03:19:44.124Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T03:19:45.250Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T03:19:45.250Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:19:45.250Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j0729472-svc
time=2026-10-24T03:21:51.429Z level=INFO msg="Cluster provisioning state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade state=Succeeded
time=2026-10-24T03:21:51.429Z level=INFO msg="ARM operation complete" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary elapsed=2m6s
time=2026-10-24T03:21:51.429Z level=INFO msg="Step 1/9: Ensuring MISE ext-authz ConfigMap" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:51.773Z level=INFO msg="ConfigMap created" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade/revision-configmap name=istio-shared-configmap-asm-1-30
time=2026-10-24T03:21:51.773Z level=INFO msg="Step 2/9: Checking tag-based namespace safety" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:51.773Z level=INFO msg="Step 3/9: Verifying control plane health and flipping tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:52.375Z level=INFO msg="Created revision tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade/revision-tag webhook=istio-revision-tag-prod-stable-aks-istio-system revision=asm-1-30
time=2026-10-24T03:21:52.375Z level=INFO msg="Step 4/9: Ensuring ingress gateway annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:52.438Z level=INFO msg="Applying annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade/ingress-annotations service=aks-istio-ingressgateway-external
time=2026-10-24T03:21:52.730Z level=INFO msg="Step 5/9: Migrating workloads to target revision" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:52.857Z level=INFO msg="Step 6/9: Running health check" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Health check passed -- checking for orphaned workloads before completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Step 7/9: Running orphan guard" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="No orphaned workloads -- completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Step 8/9: Completing canary via ARM (point of no return)" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=complete-canary cluster=ci01-j0729472-svc
time=2026-10-24T03:36:56.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-24T02:11:05Z
  Severity:    Sev3
  Labels:      alertname="ClusterCredentialMetricMissing", cluster="ci01-j4368896-svc", severity="warning"
  Description: Credential refresher expiration metric is missing for service cluster ci01-j4368896-svc.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:250]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xz7cz7"
Unexpected error:
    <*errors.joinError | 0x251e30c232d8>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc": nodes "r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x251e2f894e60>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\": nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x251e2f894de0>{
                    msg: "failed to label node r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\": nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x251e2f894dc0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\": nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x251e2fba9180>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:250]: failed to verify cluster operators on cluster "cluster-candidate-5-0-xz7cz7"
Unexpected error:
    <*errors.joinError | 0x251e30c232d8>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc": nodes "r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x251e2f894e60>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\": nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x251e2f894de0>{
                    msg: "failed to label node r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\": nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x251e2f894dc0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\": nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x251e2fba9180>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "r3d9q3y4d1r6m9l-np-xz7cz7-4flsb-cc9qc",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x20a8e1c0eb40>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9233664-mgmt-1/ocm-arohcpci01-2tks36bjou1ubb8nv8hnufkiqucajqrv-e4f0v9j4v7w6l2h: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9233664-mgmt-1/ocm-arohcpci01-2tks36bjou1ubb8nv8hnufkiqucajqrv-e4f0v9j4v7w6l2h: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
        err: <*verifiers.pollObservationError | 0x20a8e1c0eb20>{
            observation: <*errors.errorString | 0x20a8e1fc7a50>{
                s: "dev/westus3/ci01-j9233664-mgmt-1/ocm-arohcpci01-2tks36bjou1ubb8nv8hnufkiqucajqrv-e4f0v9j4v7w6l2h: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
            },
            interruption: <*fmt.wrapError | 0x20a8e1c0eb00>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T01:52:41.1693145Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x20a8e189aaf0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x20a8e1c0eae0>{
                        msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T01:52:41.1693145Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x20a8e31d5b60>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x20a8e1c0eb40>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9233664-mgmt-1/ocm-arohcpci01-2tks36bjou1ubb8nv8hnufkiqucajqrv-e4f0v9j4v7w6l2h: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j9233664-mgmt-1/ocm-arohcpci01-2tks36bjou1ubb8nv8hnufkiqucajqrv-e4f0v9j4v7w6l2h: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
        err: <*verifiers.pollObservationError | 0x20a8e1c0eb20>{
            observation: <*errors.errorString | 0x20a8e1fc7a50>{
                s: "dev/westus3/ci01-j9233664-mgmt-1/ocm-arohcpci01-2tks36bjou1ubb8nv8hnufkiqucajqrv-e4f0v9j4v7w6l2h: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
            },
            interruption: <*fmt.wrapError | 0x20a8e1c0eb00>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T01:52:41.1693145Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x20a8e189aaf0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x20a8e1c0eae0>{
                        msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-9qlmbzgtx2hq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-24T01:52:41.1693145Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x20a8e31d5b60>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19f71c1da380>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-frlx5lsjl64c" to finish creating: GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945",
      "name": "6eb737a8-b808-43b4-bafc-d691d15a4945",
      "status": "Failed",
      "startTime": "2026-10-24T02:13:35.7654536Z",
      "endTime": "2026-10-24T02:32:39.313667809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
            correlation ID: b38b110a-9400-47b0-9d30-29c40b6745e7
            client request ID: 160823fa-0861-4cd5-b970-f7ba5cee199c
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19f71c1da380>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-frlx5lsjl64c" to finish creating: GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945",
      "name": "6eb737a8-b808-43b4-bafc-d691d15a4945",
      "status": "Failed",
      "startTime": "2026-10-24T02:13:35.7654536Z",
      "endTime": "2026-10-24T02:32:39.313667809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
            correlation ID: b38b110a-9400-47b0-9d30-29c40b6745e7
            client request ID: 160823fa-0861-4cd5-b970-f7ba5cee199c

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.009s.
Expected success, but got an error:
    <*errors.errorString | 0x2de7945ac470>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [acm-agent]; observed counts: maestro=1491, hypershift=11519, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [acm-agent]; observed counts: maestro=1491, hypershift=11519, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.009s.
Expected success, but got an error:
    <*errors.errorString | 0x2de7945ac470>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [acm-agent]; observed counts: maestro=1491, hypershift=11519, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per source, missing: [acm-agent]; observed counts: maestro=1491, hypershift=11519, acm-agent=0",
    }

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-24T02:00:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2931584-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.79:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2931584/providers/microsoft.monitor/accounts/services-j2931584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-kftnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="71b5d22f-cbab-4867-b020-104124d57ca6"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-kftnj (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-24T02:01:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2931584-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2931584/providers/microsoft.monitor/accounts/services-j2931584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-w7rhw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a1d67f6c-7471-4c26-92ba-168a6c18274b"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-w7rhw (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-24T02:03:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2931584-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.79:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2931584/providers/microsoft.monitor/accounts/services-j2931584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-kftnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="71b5d22f-cbab-4867-b020-104124d57ca6"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-kftnj (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-24T02:03:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2931584-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2931584/providers/microsoft.monitor/accounts/services-j2931584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7c96c6b89d-w7rhw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a1d67f6c-7471-4c26-92ba-168a6c18274b"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7c96c6b89d-w7rhw (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xda86742c750>: 
    VerifyNodesReady failed: there are not ready nodes: cilium-cluster-cilium-np-kph42-vfrqk (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-10-24T02:36:22Z)
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fwbfcwb579g7" to finish creating: GET https://rp.j3082112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/40c4c34c-751c-413b-8e7b-4d8aef1fbbba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/40c4c34c-751c-413b-8e7b-4d8aef1fbbba",
      "name": "40c4c34c-751c-413b-8e7b-4d8aef1fbbba",
      "status": "Failed",
      "startTime": "2026-10-24T02:18:41.838217414Z",
      "endTime": "2026-10-24T02:37:46.632619034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-kph42-vfrqk: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-kph42-vfrqk: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3082112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/40c4c34c-751c-413b-8e7b-4d8aef1fbbba
            correlation ID: 260f62ef-02cf-4e9c-83e2-0f2095de0a22
            client request ID: 0ea1151f-ae49-4a4c-83b3-a2443bb08376
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xda86742c750>: 
    VerifyNodesReady failed: there are not ready nodes: cilium-cluster-cilium-np-kph42-vfrqk (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-10-24T02:36:22Z)
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fwbfcwb579g7" to finish creating: GET https://rp.j3082112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/40c4c34c-751c-413b-8e7b-4d8aef1fbbba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/40c4c34c-751c-413b-8e7b-4d8aef1fbbba",
      "name": "40c4c34c-751c-413b-8e7b-4d8aef1fbbba",
      "status": "Failed",
      "startTime": "2026-10-24T02:18:41.838217414Z",
      "endTime": "2026-10-24T02:37:46.632619034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-kph42-vfrqk: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-kph42-vfrqk: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3082112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/40c4c34c-751c-413b-8e7b-4d8aef1fbbba
            correlation ID: 260f62ef-02cf-4e9c-83e2-0f2095de0a22
            client request ID: 0ea1151f-ae49-4a4c-83b3-a2443bb08376
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-24T01:56:07Z
  Ended:       2026-10-24T02:15:08Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0562432-svc", component="backend", name="csstatedump", region="centralus", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-24T01:57:06Z
  Ended:       2026-10-24T02:15:08Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0562432-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.