CIHealth
<
Single Day: Oct 23▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 14m 1s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 eastus2 e2e failed n/a 12
e2e (12)
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
23:38:10 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0x1a3727108180>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1a37266c48a0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1a3726894020>{
                    msg: "DNS for route host agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1a3725d94d20>{
                        msg: "DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
    <*errors.joinError | 0x1a3727108180>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x1a37266c48a0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x1a3726894020>{
                    msg: "DNS for route host agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x1a3725d94d20>{
                        msg: "DNS for agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9dwkh.apps.aro.x6z1d9t4s8t4n5n.sead.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
23:38:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-5ltfgj"
Unexpected error:
    <*fmt.wrapError | 0x1c3759b85ac0>: 
    DNS for route host agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1c3759b840a0>{
            msg: "DNS for agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-5ltfgj"
Unexpected error:
    <*fmt.wrapError | 0x1c3759b85ac0>: 
    DNS for route host agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1c3759b840a0>{
            msg: "DNS for agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dq8j6.apps.aro.s6s7c3r4z7y9i8e.gjrq.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
23:38:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-5p9hrv"
Unexpected error:
    <*fmt.wrapError | 0x359991ca6200>: 
    DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x359991cc83e0>{
            msg: "DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-5p9hrv"
Unexpected error:
    <*fmt.wrapError | 0x359991ca6200>: 
    DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x359991cc83e0>{
            msg: "DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
23:38:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-wnz6kv"
Unexpected error:
    <*fmt.wrapError | 0x1aca60620e60>: 
    DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1aca604dcae0>{
            msg: "DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-wnz6kv"
Unexpected error:
    <*fmt.wrapError | 0x1aca60620e60>: 
    DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x1aca604dcae0>{
            msg: "DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
23:38:10 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x2d202c117968>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x2d202bb0ece0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
                err: <*fmt.wrapError | 0x2d202ad36960>{
                    msg: "DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
                    err: <*fmt.wrapError | 0x2d202a99c2c0>{
                        msg: "DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x2d202c117968>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x2d202bb0ece0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
                err: <*fmt.wrapError | 0x2d202ad36960>{
                    msg: "DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
                    err: <*fmt.wrapError | 0x2d202a99c2c0>{
                        msg: "DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
23:38:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-vw82ff"
Unexpected error:
    <*fmt.wrapError | 0x13ad1fd46ea0>: 
    DNS for route host agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x13ad1fb24980>{
            msg: "DNS for agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-vw82ff"
Unexpected error:
    <*fmt.wrapError | 0x13ad1fd46ea0>: 
    DNS for route host agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x13ad1fb24980>{
            msg: "DNS for agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-fxqbn.apps.aro.c7f7b4c1b4z0q6b.dcr4.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
23:38:10 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
    <*url.Error | 0x20ec8dc8cd20>: 
    Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
    {
        Op: "Get",
        URL: "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
        Err: <*net.OpError | 0x20ec8bc10690>{
            Op: "dial",
            Net: "tcp",
            Source: nil,
            Addr: nil,
            Err: <*net.DNSError | 0x20ec8bc103c0>{
                UnwrapErr: nil,
                Err: "no such host",
                Name: "console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
                Server: "172.30.0.10:53",
                IsTimeout: false,
                IsTemporary: false,
                IsNotFound: true,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
    <*url.Error | 0x20ec8dc8cd20>: 
    Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
    {
        Op: "Get",
        URL: "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
        Err: <*net.OpError | 0x20ec8bc10690>{
            Op: "dial",
            Net: "tcp",
            Source: nil,
            Addr: nil,
            Err: <*net.DNSError | 0x20ec8bc103c0>{
                UnwrapErr: nil,
                Err: "no such host",
                Name: "console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
                Server: "172.30.0.10:53",
                IsTimeout: false,
                IsTemporary: false,
                IsNotFound: true,
            },
        },
    }
occurred
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
23:38:10 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x3c147812a9c0>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c1477af35c0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x3c14791742c0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x3c14779bc3c0>{
                        msg: "DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x3c147812a9c0>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0x3c1477af35c0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0x3c14791742c0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0x3c14779bc3c0>{
                        msg: "DNS for agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-j4nk2.apps.aro.agg-cluster.as13.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
23:38:10 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xdf1d3a48c78>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0xdf1d48b58c0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0xdf1d48b46a0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0xdf1d394a3a0>{
                        msg: "DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xdf1d3a48c78>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        errs: [
            <*fmt.wrapError | 0xdf1d48b58c0>{
                msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                err: <*fmt.wrapError | 0xdf1d48b46a0>{
                    msg: "DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                    err: <*fmt.wrapError | 0xdf1d394a3a0>{
                        msg: "DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
                        err: <context.deadlineExceededError>{},
                    },
                },
            },
        ],
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
23:38:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-pfk2tm"
Unexpected error:
    <*fmt.wrapError | 0x3c89b108af80>: 
    DNS for route host agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3c89b2105000>{
            msg: "DNS for agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-pfk2tm"
Unexpected error:
    <*fmt.wrapError | 0x3c89b108af80>: 
    DNS for route host agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3c89b2105000>{
            msg: "DNS for agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vbf5l.apps.aro.a2i2k8y4h2i2q2z.3d9a.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
DNS for route host agnhost-e2e-sample-app-<id>
23:38:10 UTC · failed at e2e · test ARO-HCP should serve the default ingress certificate for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
    <*fmt.wrapError | 0x2aeb303488e0>: 
    DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x2aeb303488c0>{
            msg: "DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
    <*fmt.wrapError | 0x2aeb303488e0>: 
    DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x2aeb303488c0>{
            msg: "DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
23:38:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-q5x22b"
Unexpected error:
    <*fmt.wrapError | 0x21a3a7aea6e0>: 
    DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x21a3a7548980>{
            msg: "DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-q5x22b"
Unexpected error:
    <*fmt.wrapError | 0x21a3a7aea6e0>: 
    DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x21a3a7548980>{
            msg: "DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
1h 39m 58s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 canadacentral e2e failed n/a 10
e2e (10)
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:02 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x48191e04b0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x48191e04b0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:02 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2d5e4d930a10>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2d5e4d930a10>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyBasicAccess failed: failed to list services: Unauthorized
23:00:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-2jscs4" is viable
Unexpected error:
    <*errors.joinError | 0x1709fb679350>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x1709fb856180>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x1709fb856160>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x1709fb40e8c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x1709fb7e92c0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x1709fb7e92a0>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x1709fb8c7c20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-2jscs4" is viable
Unexpected error:
    <*errors.joinError | 0x1709fb679350>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x1709fb856180>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x1709fb856160>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x1709fb40e8c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x1709fb7e92c0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x1709fb7e92a0>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x1709fb8c7c20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
23:00:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-20-47zdm4"
Expected success, but got an error:
    <*fmt.wrapError | 0x2a6cf3f90c60>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2a6cf3f90c40>{
            observation: <*errors.errorString | 0x2a6cf562de70>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2a6cf3f90ba0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-20-47zdm4"
Expected success, but got an error:
    <*fmt.wrapError | 0x2a6cf3f90c60>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x2a6cf3f90c40>{
            observation: <*errors.errorString | 0x2a6cf562de70>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x2a6cf3f90ba0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
23:00:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-z69wtq"
Expected success, but got an error:
    <*fmt.wrapError | 0x2c961c50900>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
        err: <*fmt.wrapError | 0x2c961c506a0>{
            msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
            err: <*fmt.wrapError | 0x2c961c505e0>{
                msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
                err: <*errors.errorString | 0x2c963abc0c0>{
                    s: "rate: Wait(n=1) would exceed context deadline",
                },
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-z69wtq"
Expected success, but got an error:
    <*fmt.wrapError | 0x2c961c50900>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
        err: <*fmt.wrapError | 0x2c961c506a0>{
            msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
            err: <*fmt.wrapError | 0x2c961c505e0>{
                msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
                err: <*errors.errorString | 0x2c963abc0c0>{
                    s: "rate: Wait(n=1) would exceed context deadline",
                },
            },
        },
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
23:00:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-v65qkr"
Expected success, but got an error:
    <*fmt.wrapError | 0xbd03a06ce0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0xbd03a06ca0>{
            observation: <*errors.errorString | 0xbd05b0abd0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0xbd03a06c60>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-v65qkr"
Expected success, but got an error:
    <*fmt.wrapError | 0xbd03a06ce0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0xbd03a06ca0>{
            observation: <*errors.errorString | 0xbd05b0abd0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0xbd03a06c60>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:02 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xcc47ebfda50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xcc47ebfda50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:02 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x37eb02889a50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x37eb02889a50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
23:00:02 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2c66d40de440>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2c66d40de440>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
23:00:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-21-hwnzpg" is viable
Unexpected error:
    <*errors.joinError | 0x2be9a2dba918>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2be9a2235d00>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2be9a2235cc0>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2be9a29e6c80>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2be9a21cc080>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2be9a21cc020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2be9a3120140>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-21-hwnzpg" is viable
Unexpected error:
    <*errors.joinError | 0x2be9a2dba918>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2be9a2235d00>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2be9a2235cc0>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2be9a29e6c80>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2be9a21cc080>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2be9a21cc020>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2be9a3120140>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
2h 26m 10s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westeurope n/a passed n/a 0 n/a
2h 24m 16s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth n/a passed n/a 0 n/a
2h 20m 35s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 eastus2 e2e failed n/a 9
e2e (9)
DNS for route host agnhost-e2e-sample-app-<id>
20:07:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:526]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
    <*fmt.wrapError | 0x33122d4ce1c0>: 
    DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x33122d4ce1a0>{
            msg: "DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:526]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
    <*fmt.wrapError | 0x33122d4ce1c0>: 
    DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x33122d4ce1a0>{
            msg: "DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
DNS for route host agnhost-e2e-sample-app-lpqxs
20:07:29 UTC · failed at e2e · test ARO-HCP should serve the default ingress certificate for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
    <*fmt.wrapError | 0x398579d77aa0>: 
    DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x398579d77a60>{
            msg: "DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
    <*fmt.wrapError | 0x398579d77aa0>: 
    DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x398579d77a60>{
            msg: "DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
20:07:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-b48sqx"
Unexpected error:
    <*fmt.wrapError | 0x3e8c038e83a0>: 
    DNS for route host agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3e8c03dd0340>{
            msg: "DNS for agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-b48sqx"
Unexpected error:
    <*fmt.wrapError | 0x3e8c038e83a0>: 
    DNS for route host agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3e8c03dd0340>{
            msg: "DNS for agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dgfpk.apps.aro.c2c1j7v6o7x9e3w.no9b.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
20:07:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-hb5rp9"
Unexpected error:
    <*fmt.wrapError | 0x38ad847b15a0>: 
    DNS for route host agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x38ad8403b0a0>{
            msg: "DNS for agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-hb5rp9"
Unexpected error:
    <*fmt.wrapError | 0x38ad847b15a0>: 
    DNS for route host agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x38ad8403b0a0>{
            msg: "DNS for agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nplh9.apps.aro.p4i4z6i9y0h8m3b.r4vt.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
20:07:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-9d6fpw"
Unexpected error:
    <*fmt.wrapError | 0x21815df3d960>: 
    DNS for route host agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x21815eb44080>{
            msg: "DNS for agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-9d6fpw"
Unexpected error:
    <*fmt.wrapError | 0x21815df3d960>: 
    DNS for route host agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x21815eb44080>{
            msg: "DNS for agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-67lsh.apps.aro.i0t9o8c4q5f8r2m.4m1k.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True
20:07:29 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0x397e13438ac8>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    {
        errs: [
            <context.deadlineExceededError>{},
            <*errors.errorString | 0x397e134eabf0>{
                s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0x397e13438ac8>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    {
        errs: [
            <context.deadlineExceededError>{},
            <*errors.errorString | 0x397e134eabf0>{
                s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
            },
        ],
    }
occurred
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
20:07:29 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 634.146s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: [
        "console (Available=False)",
        "ingress (Available=False)",
    ]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 634.146s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: [
        "console (Available=False)",
        "ingress (Available=False)",
    ]
to be empty
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
20:07:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-mx6tbf"
Unexpected error:
    <*fmt.wrapError | 0x10452515d3c0>: 
    DNS for route host agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1045256005e0>{
            msg: "DNS for agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-mx6tbf"
Unexpected error:
    <*fmt.wrapError | 0x10452515d3c0>: 
    DNS for route host agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1045256005e0>{
            msg: "DNS for agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p2dgw.apps.aro.g8l5h2e8g9b9a4u.ex01.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
20:07:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-fvww2g"
Unexpected error:
    <*fmt.wrapError | 0x194223a493c0>: 
    DNS for route host agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x194223dd6660>{
            msg: "DNS for agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-fvww2g"
Unexpected error:
    <*fmt.wrapError | 0x194223a493c0>: 
    DNS for route host agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x194223dd6660>{
            msg: "DNS for agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wzr8t.apps.aro.t9b6r2z8e5x9w7g.5hae.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
2h 2m 40s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 switzerlandnorth n/a passed n/a 0 n/a
2h 4m 55s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 switzerlandnorth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
14:34:17 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x9a8a07812a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-npbbgrcnjvjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
      "name": "5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
      "status": "Failed",
      "startTime": "2026-10-23T14:54:47.282384682Z",
      "endTime": "2026-10-23T15:13:53.224517483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
            correlation ID: 90989f90-f502-41d8-955e-974a04d1e9d9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x9a8a07812a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-npbbgrcnjvjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
      "name": "5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
      "status": "Failed",
      "startTime": "2026-10-23T14:54:47.282384682Z",
      "endTime": "2026-10-23T15:13:53.224517483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
            correlation ID: 90989f90-f502-41d8-955e-974a04d1e9d9
            client request ID: unavailable
occurred
2h 15m 47s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 centralindia n/a passed n/a 0 n/a
2h 16m 53s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast n/a passed n/a 0 n/a
2h 16m 29s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 switzerlandnorth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist; provider Microsoft.RedHatOpenShift
11:31:42 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0x2ef724ca1cc0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-gnnvxwc94fcv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
      "name": "eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
      "status": "Failed",
      "startTime": "2026-10-23T13:03:20.324115794Z",
      "endTime": "2026-10-23T13:22:20.156214194Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
            correlation ID: 5b15bb34-6e8a-47a7-ac3b-f4e7ecb8ef82
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0x2ef724ca1cc0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-gnnvxwc94fcv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
      "name": "eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
      "status": "Failed",
      "startTime": "2026-10-23T13:03:20.324115794Z",
      "endTime": "2026-10-23T13:22:20.156214194Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
            correlation ID: 5b15bb34-6e8a-47a7-ac3b-f4e7ecb8ef82
            client request ID: unavailable
occurred
2h 16m 46s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 canadacentral n/a passed n/a 0 n/a
2h 16m 18s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a
2h 6m 38s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast e2e failed n/a 10
e2e (10)
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2b0e7254f60>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2b0e7254f60>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xc1401787740>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xc1401787740>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x24769a10c7f0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x24769a10c7f0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
02:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-8zbqmf"
Expected success, but got an error:
    <*fmt.wrapError | 0x15ca294921a0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x15ca29492180>{
            observation: <*errors.errorString | 0x15ca28eb6740>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x15ca294920a0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-8zbqmf"
Expected success, but got an error:
    <*fmt.wrapError | 0x15ca294921a0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x15ca29492180>{
            observation: <*errors.errorString | 0x15ca28eb6740>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x15ca294920a0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
02:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-22-dppnl7"
Expected success, but got an error:
    <*fmt.wrapError | 0x254a74952740>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x254a74952660>{
            observation: <*errors.errorString | 0x254a747beb00>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x254a74952640>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-22-dppnl7"
Expected success, but got an error:
    <*fmt.wrapError | 0x254a74952740>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x254a74952660>{
            observation: <*errors.errorString | 0x254a747beb00>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x254a74952640>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x1ee734123b50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x1ee734123b50>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
02:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-drlc89"
Expected success, but got an error:
    <*fmt.wrapError | 0x3bc58413f6c0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x3bc584326750>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-drlc89"
Expected success, but got an error:
    <*fmt.wrapError | 0x3bc58413f6c0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x3bc584326750>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
02:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-21-b4mmvn"
Expected success, but got an error:
    <*fmt.wrapError | 0x123062b14220>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x123062b141a0>{
            observation: <*errors.errorString | 0x1230626cb5f0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x123062b14140>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-21-b4mmvn"
Expected success, but got an error:
    <*fmt.wrapError | 0x123062b14220>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x123062b141a0>{
            observation: <*errors.errorString | 0x1230626cb5f0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x123062b14140>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:29 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x1b6937cc7c90>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x1b6937cc7c90>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
02:00:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-gqg8kx"
Expected success, but got an error:
    <*fmt.wrapError | 0x3c86c376e000>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x3c86c4fb96a0>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-gqg8kx"
Expected success, but got an error:
    <*fmt.wrapError | 0x3c86c376e000>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x3c86c4fb96a0>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
30m 1s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)