CIHealth
<
Single Day: May 20▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 10
e2e (10)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
23:07:55 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ca6168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-7tpw8l" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ca6168>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-7tpw8l" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
Deserializaion Error: no output from command
23:07:55 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
ing specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request (Spec Runtime: 2h29m59.885s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44
        In [It] (Node Runtime: 2h29m59.885s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44
          At [By Step] creating the HCP cluster (Step Runtime: 31m42.234s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:76

          Spec Goroutine
          goroutine 116 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x90e87c8, 0xc000daac30}, 0x85624cf?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x91280a0, {0x90e87c8, 0xc000daac30}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x90e8838?, 0xc000b193b0?}, {{0x9102420?, 0xc00100a280?}, 0xc001989050?}, 0xc00008f750, {0xc000966420, 0x11}, {0x85827de, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:850
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0xc000b009e7, 0x7}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346
          > github.com/Azure/ARO-HCP/test/e2e.init.func15.1.1({0x7f6e2c6e7860, 0xc001016810})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:77
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x9102a30?, 0xc001016810?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 114
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84 @ 05/21/26 01:52:31.235
  "ts"="2026-05-21 01:52:31.236130" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-05-21 01:52:31.236193" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-name-cluster-s9mrsr" "resourceGroup"="patch-name-6glw6m"
  "ts"="2026-05-21 01:52:31.505346" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-name-cluster-s9mrsr" "resourceGroup"=""
  "ts"="2026-05-21 01:52:32.976416" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-edcc54feb78eb6f23e76e41140aef2c970127915770a00eb26db2362637d7984.yaml.gz"
  "ts"="2026-05-21 01:52:32.976513" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-edcc54feb78eb6f23e76e41140aef2c970127915770a00eb26db2362637d7984.yaml.gz"
  "ts"="2026-05-21 01:52:32.976597" "level"=0 "msg"="collecting debug info"
  "ts"="2026-05-21 01:52:32.978037" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-40"
  "ts"="2026-05-21 01:52:32.978051" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-name-6glw6m"
  "ts"="2026-05-21 01:52:33.371789" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-40" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-05-21 01:52:33.434158" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-name-6glw6m" "deployment"="pooledNonMsiScopedAssignments-rr7tdjqoygobu"
  "ts"="2026-05-21 01:52:33.487353" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-40" "deployment"="msi-deployment-40"
  "ts"="2026-05-21 01:52:33.673654" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-name-6glw6m" "deployment"="customer-infra-patch-name-cluster-s9mrsr"
  "ts"="2026-05-21 01:52:33.945032" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-05-21 01:52:34.176123" "level"=0 "msg"="deleting created resources"
  "ts"="2026-05-21 01:52:34.176366" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-name-6glw6m"
Command Error: signal: killed
Deserializaion Error: no output from command
ing specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request (Spec Runtime: 2h29m59.885s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44
        In [It] (Node Runtime: 2h29m59.885s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44
          At [By Step] creating the HCP cluster (Step Runtime: 31m42.234s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:76

          Spec Goroutine
          goroutine 116 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x90e87c8, 0xc000daac30}, 0x85624cf?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x91280a0, {0x90e87c8, 0xc000daac30}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x90e8838?, 0xc000b193b0?}, {{0x9102420?, 0xc00100a280?}, 0xc001989050?}, 0xc00008f750, {0xc000966420, 0x11}, {0x85827de, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:850
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0xc000b009e7, 0x7}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346
          > github.com/Azure/ARO-HCP/test/e2e.init.func15.1.1({0x7f6e2c6e7860, 0xc001016810})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:77
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x9102a30?, 0xc001016810?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 114
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84 @ 05/21/26 01:52:31.235
  "ts"="2026-05-21 01:52:31.236130" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-05-21 01:52:31.236193" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-name-cluster-s9mrsr" "resourceGroup"="patch-name-6glw6m"
  "ts"="2026-05-21 01:52:31.505346" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-name-cluster-s9mrsr" "resourceGroup"=""
  "ts"="2026-05-21 01:52:32.976416" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-edcc54feb78eb6f23e76e41140aef2c970127915770a00eb26db2362637d7984.yaml.gz"
  "ts"="2026-05-21 01:52:32.976513" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-edcc54feb78eb6f23e76e41140aef2c970127915770a00eb26db2362637d7984.yaml.gz"
  "ts"="2026-05-21 01:52:32.976597" "level"=0 "msg"="collecting debug info"
  "ts"="2026-05-21 01:52:32.978037" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-40"
  "ts"="2026-05-21 01:52:32.978051" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-name-6glw6m"
  "ts"="2026-05-21 01:52:33.371789" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-40" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-05-21 01:52:33.434158" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-name-6glw6m" "deployment"="pooledNonMsiScopedAssignments-rr7tdjqoygobu"
  "ts"="2026-05-21 01:52:33.487353" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-40" "deployment"="msi-deployment-40"
  "ts"="2026-05-21 01:52:33.673654" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-name-6glw6m" "deployment"="customer-infra-patch-name-cluster-s9mrsr"
  "ts"="2026-05-21 01:52:33.945032" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-05-21 01:52:34.176123" "level"=0 "msg"="deleting created resources"
  "ts"="2026-05-21 01:52:34.176366" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-name-6glw6m"
Command Error: signal: killed
Deserializaion Error: no output from command
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
23:07:55 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001674318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bpxgsp" in resourcegroup="rg-candidate-4-20-vzzzlh" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001674318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bpxgsp" in resourcegroup="rg-candidate-4-20-vzzzlh" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:07:55 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015895c0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-w8tvcz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-w8tvcz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015895c0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-w8tvcz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-w8tvcz, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
23:07:55 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000b9c438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-9v22th" in resourcegroup="rg-np-version-upgrade-9v22th-5s2l57" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000b9c438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-9v22th" in resourcegroup="rg-np-version-upgrade-9v22th-5s2l57" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
23:07:55 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f160a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7jm9fm" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f160a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7jm9fm" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
UpdateHCPCluster20251223 timed out after <minutes> minutes; context deadline exceeded
23:07:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:223]: Unexpected error:
    <*fmt.wrapErrors | 0xc000510780>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8cb4lx" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-8cb4lx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:223]: Unexpected error:
    <*fmt.wrapErrors | 0xc000510780>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8cb4lx" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-8cb4lx, error: context deadline exceeded
    ...
occurred
Interrupted by User
23:07:55 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:30]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:30]: Interrupted by User
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
23:07:55 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00145c258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-5lhhw2" in resourcegroup="rg-np-version-upgrade-5lhhw2-s7gzkl" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00145c258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-5lhhw2" in resourcegroup="rg-np-version-upgrade-5lhhw2-s7gzkl" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
Deserializaion Error: no output from command
23:07:55 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
od/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
          > github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000de0b88, {0xc000d1f1e0?, 0x0?, 0x0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
          > github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000da0008, {0x0, 0x0}, {0xc6e2ca0, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000d1f1a0, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2e18a85?, 0xc000c01ec0?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/ginkgo/util.go:84
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x90e87c8, 0xc0008b5590}, 0xc000e8b520, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:230

          goroutine 107 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000e8b520?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 106
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:119 @ 05/21/26 01:52:31.225
  "ts"="2026-05-21 01:52:31.225980" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-05-21 01:52:31.226030" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-21-6spgxp-87qlp2" "resourceGroup"="rg-candidate-4-21-rvqdlq"
  "ts"="2026-05-21 01:52:31.949212" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-21-6spgxp-87qlp2" "resourceGroup"=""
  "ts"="2026-05-21 01:52:33.901519" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-6427b6424c4038306dc527fd5f7fc0b377fe9c26e56432adfe480dff6189e0c7.yaml.gz"
  "ts"="2026-05-21 01:52:33.901684" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-6427b6424c4038306dc527fd5f7fc0b377fe9c26e56432adfe480dff6189e0c7.yaml.gz"
  "ts"="2026-05-21 01:52:33.901758" "level"=0 "msg"="collecting debug info"
  "ts"="2026-05-21 01:52:33.903207" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-67"
  "ts"="2026-05-21 01:52:33.903226" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-21-rvqdlq"
  "ts"="2026-05-21 01:52:34.245138" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-67" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-05-21 01:52:34.297415" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-21-rvqdlq" "deployment"="pooledNonMsiScopedAssignments-txlkb6a3h2s3w"
  "ts"="2026-05-21 01:52:34.355759" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-67" "deployment"="msi-deployment-67"
  "ts"="2026-05-21 01:52:34.536073" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-21-rvqdlq" "deployment"="customer-infra-cluster-candidate-4-21-6spgxp-87qlp2"
  "ts"="2026-05-21 01:52:34.830797" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-05-21 01:52:35.321929" "level"=0 "msg"="deleting created resources"
  "ts"="2026-05-21 01:52:35.322167" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-21-rvqdlq"
Command Error: signal: killed
Deserializaion Error: no output from command
od/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
          > github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000de0b88, {0xc000d1f1e0?, 0x0?, 0x0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
          > github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000da0008, {0x0, 0x0}, {0xc6e2ca0, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000d1f1a0, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2e18a85?, 0xc000c01ec0?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/ginkgo/util.go:84
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x90e87c8, 0xc0008b5590}, 0xc000e8b520, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:230

          goroutine 107 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000e8b520?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 106
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:119 @ 05/21/26 01:52:31.225
  "ts"="2026-05-21 01:52:31.225980" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-05-21 01:52:31.226030" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-21-6spgxp-87qlp2" "resourceGroup"="rg-candidate-4-21-rvqdlq"
  "ts"="2026-05-21 01:52:31.949212" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-21-6spgxp-87qlp2" "resourceGroup"=""
  "ts"="2026-05-21 01:52:33.901519" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-6427b6424c4038306dc527fd5f7fc0b377fe9c26e56432adfe480dff6189e0c7.yaml.gz"
  "ts"="2026-05-21 01:52:33.901684" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-6427b6424c4038306dc527fd5f7fc0b377fe9c26e56432adfe480dff6189e0c7.yaml.gz"
  "ts"="2026-05-21 01:52:33.901758" "level"=0 "msg"="collecting debug info"
  "ts"="2026-05-21 01:52:33.903207" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-67"
  "ts"="2026-05-21 01:52:33.903226" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-21-rvqdlq"
  "ts"="2026-05-21 01:52:34.245138" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-67" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-05-21 01:52:34.297415" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-21-rvqdlq" "deployment"="pooledNonMsiScopedAssignments-txlkb6a3h2s3w"
  "ts"="2026-05-21 01:52:34.355759" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-67" "deployment"="msi-deployment-67"
  "ts"="2026-05-21 01:52:34.536073" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-21-rvqdlq" "deployment"="customer-infra-cluster-candidate-4-21-6spgxp-87qlp2"
  "ts"="2026-05-21 01:52:34.830797" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-05-21 01:52:35.321929" "level"=0 "msg"="deleting created resources"
  "ts"="2026-05-21 01:52:35.322167" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-21-rvqdlq"
Command Error: signal: killed
Deserializaion Error: no output from command
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
tls: failed to verify certificate: x509: certificate signed by unknown authority
20:02:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: Unexpected error:
    <*fmt.wrapError | 0xc00095e660>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-69q6l.apps.aro.v3s6i0y3r3r2s1o.hzrs.switzerlandnorth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: Unexpected error:
    <*fmt.wrapError | 0xc00095e660>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-69q6l.apps.aro.v3s6i0y3r3r2s1o.hzrs.switzerlandnorth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
client rate limiter Wait returned an error: context deadline exceeded
05:29:32 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:207]: Unexpected error:
    <*errors.joinError | 0xc000c80540>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bqh2d5" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:207]: Unexpected error:
    <*errors.joinError | 0xc000c80540>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bqh2d5" to finish creating: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a