<
Single Day: Mar 28▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 9e3b120 · merge bf05195
n/a e2e failed ⚠#4674 (merged) 1
e2e (1)
context deadline exceeded
20:03:42 UTC · failed at e2e · test Create HCPOpenShiftCluster with Cilium CNI should create a no-CNI cluster with private etcd and install Cilium · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:173]: Unexpected error:
    <*fmt.wrapError | 0xc002a68000>: 
    failed to install cilium chart: resource not ready, name: cilium-operator, kind: Deployment, status: InProgress
    context deadline exceeded
    {
        msg: "failed to install cilium chart: resource not ready, name: cilium-operator, kind: Deployment, status: InProgress\ncontext deadline exceeded",
        err: <*errors.joinError | 0xc0011b4888>{
            errs: [
                <*errors.errorString | 0xc0022bc3d0>{
                    s: "resource not ready, name: cilium-operator, kind: Deployment, status: InProgress",
                },
                <context.deadlineExceededError>{},
            ],
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:173]: Unexpected error:
    <*fmt.wrapError | 0xc002a68000>: 
    failed to install cilium chart: resource not ready, name: cilium-operator, kind: Deployment, status: InProgress
    context deadline exceeded
    {
        msg: "failed to install cilium chart: resource not ready, name: cilium-operator, kind: Deployment, status: InProgress\ncontext deadline exceeded",
        err: <*errors.joinError | 0xc0011b4888>{
            errs: [
                <*errors.errorString | 0xc0022bc3d0>{
                    s: "resource not ready, name: cilium-operator, kind: Deployment, status: InProgress",
                },
                <context.deadlineExceededError>{},
            ],
        },
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 301d51b
n/a n/a passed #4673 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head fa52717 · merge fa52717
n/a n/a passed #4672 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head d013cfd · merge ba58288
n/a n/a passed #4671 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 1f06de0 · merge 1f06de0
n/a n/a passed #4666 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head d85b5fe
n/a n/a passed #4670 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head cebfe71 · merge cebfe71
n/a n/a passed #4669 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 120e03c
n/a provision failed ⚠#4668 (closed) 1
provision (1)
Error response from registry: failed to resolve sha256:<hex>: unauthorized: access to the requested resource is not authorized: map[]
07:33:21 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/global/mirror-image · suite step graph
Show raw failure
time=2026-03-28T08:11:14.140Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> clustersService.image.registry <nil>}:{<nil> clustersService.image.repository <nil>}@{<nil> clustersService.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-28T08:11:19.154Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build05.ci.openshift.org registry.build05.ci.openshift.org registry.build05.ci.openshift.org registry.build05.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image quay.io/raelga/aro-hcp-clusters-service@sha256:4eb6142d00b2a70ad664a70489c343653420fbef712678f2e895e9be7adf17d8 to arohcpsvcdev.azurecr.io/raelga/aro-hcp-clusters-service:4eb6142d00b2a70ad664a70489c343653420fbef712678f2e895e9be7adf17d8.\nThe image will still be available under it's original digest sha256:4eb6142d00b2a70ad664a70489c343653420fbef712678f2e895e9be7adf17d8 in the target registry.\nError response from registry: failed to resolve sha256:4eb6142d00b2a70ad664a70489c343653420fbef712678f2e895e9be7adf17d8: unauthorized: access to the requested resource is not authorized: map[]\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 809233b · merge 1f06de0
n/a e2e failed ★#4666 (merged) 4
e2e (4)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
05:12:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001060228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-5vsdzh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000c2afa0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000c2af60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000c2af20>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b37d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001060228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-5vsdzh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000c2afa0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000c2af60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000c2af20>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-5vsdzh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b37d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
05:12:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.009s.
Expected success, but got an error:
    <*errors.joinError | 0xc00101c4e0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc000b00c20>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc000804560>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.009s.
Expected success, but got an error:
    <*errors.joinError | 0xc00101c4e0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc000b00c20>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc000804560>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
05:12:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc00100f2f0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc000956ee0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc0012f65e0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc00100f2f0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc000956ee0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc0012f65e0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
05:12:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0013cc180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-j9fds8" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000b0df20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000b0df00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000b0dee0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b37d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0013cc180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-j9fds8" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000b0df20>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000b0df00>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000b0dee0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-j9fds8\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b37d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head ea1e304
n/a n/a passed #4639 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 965139d
n/a e2e failed ★#4665 (closed) 3
e2e (3)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
03:16:29 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00017e150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-8s5fww" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000969680>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000969660>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000969640>{
                        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b17d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00017e150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-8s5fww" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000969680>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000969660>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000969640>{
                        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8s5fww\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b17d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
03:16:29 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0013a6570>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc001449840>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc0007bce30>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0013a6570>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc001449840>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc0007bce30>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
03:16:29 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d42438>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc000a81540>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc00073b050>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d42438>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc000a81540>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc00073b050>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 809233b · merge 1f06de0
n/a e2e failed ★#4666 (merged) 3
e2e (3)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
01:33:10 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001120360>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-w2p7hf" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d8a280>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-w2p7hf\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d8a260>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-w2p7hf\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d8a240>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-w2p7hf\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b37d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001120360>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-w2p7hf" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d8a280>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-w2p7hf\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d8a260>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-w2p7hf\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d8a240>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-w2p7hf\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b37d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
01:33:10 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011fb800>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc000828580>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc000a2c320>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011fb800>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc000828580>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc000a2c320>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
01:33:10 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ffd110>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc000d81c80>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc0007aae10>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ffd110>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc000d81c80>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc0007aae10>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 965139d
n/a e2e failed ★#4665 (closed) 3
e2e (3)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
01:26:00 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001264240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lnw9t5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009b9520>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lnw9t5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009b94e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lnw9t5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009b94c0>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lnw9t5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b17d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:246] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001264240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lnw9t5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009b9520>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lnw9t5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009b94e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lnw9t5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0009b94c0>{
                        msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lnw9t5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb2b17d0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
01:26:00 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00108a5d0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc000e04620>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc000a38830>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00108a5d0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial "4.20.16"
    {
        errs: [
            <*fmt.wrapError | 0xc000e04620>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.16) failed: no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                err: <*errors.errorString | 0xc000a38830>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.16\"",
                },
            },
        ],
    }
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
01:26:00 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.009s.
Expected success, but got an error:
    <*errors.joinError | 0xc0012356b0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc0009ee6c0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc0008e03b0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:122]: Timed out after 2400.009s.
Expected success, but got an error:
    <*errors.joinError | 0xc0012356b0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial "4.21.6"
    {
        errs: [
            <*fmt.wrapError | 0xc0009ee6c0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.21.6) failed: no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                err: <*errors.errorString | 0xc0008e03b0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.21.6\"",
                },
            },
        ],
    }