CIHealth
<
Single Day: Apr 28▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 38
e2e (38)
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc001318540>: 
    failed to create HCP cluster ea-listxwwn4v, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listxwwn4v in resource group ea-list-lcwskt, error: failed waiting for cluster="ea-listxwwn4v" in resourcegroup="ea-list-lcwskt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listxwwn4v in resource group ea-list-lcwskt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc001318540>: 
    failed to create HCP cluster ea-listxwwn4v, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listxwwn4v in resource group ea-list-lcwskt, error: failed waiting for cluster="ea-listxwwn4v" in resourcegroup="ea-list-lcwskt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listxwwn4v in resource group ea-list-lcwskt, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f412f0>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-c6ns4t" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f412f0>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-c6ns4t" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc00100afc0>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-f9gbcf, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-f9gbcf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-f9gbcf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc00100afc0>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-f9gbcf, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-f9gbcf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-f9gbcf, error: context deadline exceeded
    ...
occurred
context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011e4cc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-fs2kw7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-fs2kw7 in resource group rg-np-version-upgrade-fs2kw7-npgt57, error: failed waiting for cluster="np-version-upgrade-cluster-fs2kw7" in resourcegroup="rg-np-version-upgrade-fs2kw7-npgt57" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-fs2kw7 in resource group rg-np-version-upgrade-fs2kw7-npgt57, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011e4cc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-fs2kw7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-fs2kw7 in resource group rg-np-version-upgrade-fs2kw7-npgt57, error: failed waiting for cluster="np-version-upgrade-cluster-fs2kw7" in resourcegroup="rg-np-version-upgrade-fs2kw7-npgt57" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-fs2kw7 in resource group rg-np-version-upgrade-fs2kw7-npgt57, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc001614ab0>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-cxwv92, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cxwv92" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-cxwv92, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc001614ab0>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-cxwv92, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cxwv92" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-cxwv92, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error:
    <*fmt.wrapErrors | 0xc00100ccc0>: 
    failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-hdpjnx, error: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hdpjnx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-hdpjnx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error:
    <*fmt.wrapErrors | 0xc00100ccc0>: 
    failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-hdpjnx, error: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hdpjnx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-hdpjnx, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015ccab0>: 
    failed to create HCP cluster np-version-upgrade-cluster-xscdn4, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-xscdn4 in resource group rg-np-version-upgrade-xscdn4-l9xftc, error: failed waiting for cluster="np-version-upgrade-cluster-xscdn4" in resourcegroup="rg-np-version-upgrade-xscdn4-l9xftc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-xscdn4 in resource group rg-np-version-upgrade-xscdn4-l9xftc, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015ccab0>: 
    failed to create HCP cluster np-version-upgrade-cluster-xscdn4, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-xscdn4 in resource group rg-np-version-upgrade-xscdn4-l9xftc, error: failed waiting for cluster="np-version-upgrade-cluster-xscdn4" in resourcegroup="rg-np-version-upgrade-xscdn4-l9xftc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-xscdn4 in resource group rg-np-version-upgrade-xscdn4-l9xftc, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc00135cae0>: 
    failed to create HCP cluster cluster-candidate-4-21-4m2mq8, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-21-4m2mq8 in resource group rg-candidate-4-21-zt7kdp, error: failed waiting for cluster="cluster-candidate-4-21-4m2mq8" in resourcegroup="rg-candidate-4-21-zt7kdp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-21-4m2mq8 in resource group rg-candidate-4-21-zt7kdp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc00135cae0>: 
    failed to create HCP cluster cluster-candidate-4-21-4m2mq8, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-21-4m2mq8 in resource group rg-candidate-4-21-zt7kdp, error: failed waiting for cluster="cluster-candidate-4-21-4m2mq8" in resourcegroup="rg-candidate-4-21-zt7kdp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-21-4m2mq8 in resource group rg-candidate-4-21-zt7kdp, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000911740>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-6cflnm, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-6cflnm" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-6cflnm, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000911740>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-6cflnm, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-6cflnm" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-6cflnm, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc00154e990>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-lk4qkg, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-lk4qkg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-lk4qkg, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc00154e990>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-lk4qkg, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-lk4qkg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-lk4qkg, error: context deadline exceeded
    ...
occurred
Cluster should become ready within 45 minutes
02:00:05 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: ...
to be true
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014e2900>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-vrjtg9, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-vrjtg9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-vrjtg9, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014e2900>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-vrjtg9, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-vrjtg9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-vrjtg9, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc001651050>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-nffbvf, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nffbvf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-nffbvf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc001651050>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-nffbvf, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nffbvf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-nffbvf, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: Unexpected error:
    <*fmt.wrapErrors | 0xc001517ef0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-m5bl9d" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-m5bl9d, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: Unexpected error:
    <*fmt.wrapErrors | 0xc001517ef0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-m5bl9d" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-m5bl9d, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc001507320>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-zpnfp4, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-zpnfp4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-zpnfp4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc001507320>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-zpnfp4, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-zpnfp4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-zpnfp4, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc0017962d0>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-5j4wzp, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5j4wzp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-5j4wzp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc0017962d0>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-5j4wzp, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5j4wzp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-5j4wzp, error: context deadline exceeded
    ...
occurred
Deserializaion Error: no output from command
02:00:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
ithub.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
          > github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000e8eb88, {0xc000c39020?, 0x0?, 0x0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
          > github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000d9f508, {0x0, 0x0}, {0xc284540, 0x0, 0x0}, {0xc00007c094, 0x2a}, 0x0, {0xc000c39000, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007c094, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d3ea85?, 0xc000a00000?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/ginkgo/util.go:85
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8da8ae8, 0xc000ac4aa0}, 0xc000ab3520, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:230

          goroutine 118 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000ab3520?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 117
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115 @ 04/28/26 04:31:29.581
  "ts"="2026-04-28 04:31:29.581788" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-28 04:31:29.581897" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"="rg-candidate-4-22-fr8wfb"
  "ts"="2026-04-28 04:31:29.754059" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"=""
  "ts"="2026-04-28 04:31:30.945527" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
  "ts"="2026-04-28 04:31:30.945809" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
  "ts"="2026-04-28 04:31:30.945911" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-28 04:31:30.947534" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-120"
  "ts"="2026-04-28 04:31:30.947616" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-22-fr8wfb"
  "ts"="2026-04-28 04:31:31.115549" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="pooledNonMsiScopedAssignments-dwo7nh5uecnn2"
  "ts"="2026-04-28 04:31:31.143526" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-28 04:31:31.256209" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn"
  "ts"="2026-04-28 04:31:31.642441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="msi-deployment-120"
  "ts"="2026-04-28 04:31:31.783617" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-28 04:31:32.120530" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-28 04:31:32.120780" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-22-fr8wfb"
Command Error: signal: killed
Deserializaion Error: no output from command
ithub.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
          > github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000e8eb88, {0xc000c39020?, 0x0?, 0x0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
          > github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000d9f508, {0x0, 0x0}, {0xc284540, 0x0, 0x0}, {0xc00007c094, 0x2a}, 0x0, {0xc000c39000, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007c094, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d3ea85?, 0xc000a00000?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/ginkgo/util.go:85
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8da8ae8, 0xc000ac4aa0}, 0xc000ab3520, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:230

          goroutine 118 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000ab3520?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 117
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115 @ 04/28/26 04:31:29.581
  "ts"="2026-04-28 04:31:29.581788" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-28 04:31:29.581897" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"="rg-candidate-4-22-fr8wfb"
  "ts"="2026-04-28 04:31:29.754059" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"=""
  "ts"="2026-04-28 04:31:30.945527" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
  "ts"="2026-04-28 04:31:30.945809" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
  "ts"="2026-04-28 04:31:30.945911" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-28 04:31:30.947534" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-120"
  "ts"="2026-04-28 04:31:30.947616" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-22-fr8wfb"
  "ts"="2026-04-28 04:31:31.115549" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="pooledNonMsiScopedAssignments-dwo7nh5uecnn2"
  "ts"="2026-04-28 04:31:31.143526" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-28 04:31:31.256209" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn"
  "ts"="2026-04-28 04:31:31.642441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="msi-deployment-120"
  "ts"="2026-04-28 04:31:31.783617" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-28 04:31:32.120530" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-28 04:31:32.120780" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-22-fr8wfb"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc00156a420>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-5ls4md" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-5ls4md, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc00156a420>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-5ls4md" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-5ls4md, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc000eda600>: 
    failed to create HCP cluster cluster-zstream-4-20-4nc5dt, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-4nc5dt in resource group rg-zstream-upgrade-4-20-t5nhmt, error: failed waiting for cluster="cluster-zstream-4-20-4nc5dt" in resourcegroup="rg-zstream-upgrade-4-20-t5nhmt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-4nc5dt in resource group rg-zstream-upgrade-4-20-t5nhmt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc000eda600>: 
    failed to create HCP cluster cluster-zstream-4-20-4nc5dt, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-4nc5dt in resource group rg-zstream-upgrade-4-20-t5nhmt, error: failed waiting for cluster="cluster-zstream-4-20-4nc5dt" in resourcegroup="rg-zstream-upgrade-4-20-t5nhmt" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-4nc5dt in resource group rg-zstream-upgrade-4-20-t5nhmt, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00081a600>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-xbprh4, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-xbprh4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-xbprh4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00081a600>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-xbprh4, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-xbprh4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-xbprh4, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f52ae0>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-g7ghtq, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-g7ghtq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-g7ghtq, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f52ae0>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-g7ghtq, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-g7ghtq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-g7ghtq, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010848a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-kvvsxl, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-kvvsxl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-kvvsxl, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010848a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-kvvsxl, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-kvvsxl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-kvvsxl, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc00135bec0>: 
    failed to create HCP cluster cluster-candidate-4-20-4mvl7g, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: failed waiting for cluster="cluster-candidate-4-20-4mvl7g" in resourcegroup="rg-candidate-4-20-vm7tln" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc00135bec0>: 
    failed to create HCP cluster cluster-candidate-4-20-4mvl7g, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: failed waiting for cluster="cluster-candidate-4-20-4mvl7g" in resourcegroup="rg-candidate-4-20-vm7tln" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc00147f230>: 
    failed to create HCP cluster cluster-zstream-4-21-hh88n8, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-hh88n8 in resource group rg-zstream-upgrade-4-21-jrzh8q, error: failed waiting for cluster="cluster-zstream-4-21-hh88n8" in resourcegroup="rg-zstream-upgrade-4-21-jrzh8q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-hh88n8 in resource group rg-zstream-upgrade-4-21-jrzh8q, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc00147f230>: 
    failed to create HCP cluster cluster-zstream-4-21-hh88n8, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-hh88n8 in resource group rg-zstream-upgrade-4-21-jrzh8q, error: failed waiting for cluster="cluster-zstream-4-21-hh88n8" in resourcegroup="rg-zstream-upgrade-4-21-jrzh8q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-hh88n8 in resource group rg-zstream-upgrade-4-21-jrzh8q, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc001176360>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-g4j47s, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-g4j47s" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-g4j47s, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc001176360>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-g4j47s, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-g4j47s" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-g4j47s, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:256]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a23980>: 
    failed to create HCP cluster shoebox-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-87cjmk, error: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-87cjmk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-87cjmk, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:256]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a23980>: 
    failed to create HCP cluster shoebox-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-87cjmk, error: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-87cjmk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-87cjmk, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc000862bd0>: 
    failed to create HCP cluster np-version-upgrade-cluster-qcxn5c, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qcxn5c in resource group rg-np-version-upgrade-qcxn5c-22qsq6, error: failed waiting for cluster="np-version-upgrade-cluster-qcxn5c" in resourcegroup="rg-np-version-upgrade-qcxn5c-22qsq6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qcxn5c in resource group rg-np-version-upgrade-qcxn5c-22qsq6, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:130]: Unexpected error:
    <*fmt.wrapErrors | 0xc000862bd0>: 
    failed to create HCP cluster np-version-upgrade-cluster-qcxn5c, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qcxn5c in resource group rg-np-version-upgrade-qcxn5c-22qsq6, error: failed waiting for cluster="np-version-upgrade-cluster-qcxn5c" in resourcegroup="rg-np-version-upgrade-qcxn5c-22qsq6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-qcxn5c in resource group rg-np-version-upgrade-qcxn5c-22qsq6, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: Unexpected error:
    <*fmt.wrapErrors | 0xc00113c630>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-mk5cms" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: Unexpected error:
    <*fmt.wrapErrors | 0xc00113c630>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-mk5cms" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ece660>: 
    failed to create HCP cluster cluster-zstream-4-19-l2jj6j, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-l2jj6j in resource group rg-zstream-upgrade-4-19-hs9zt7, error: failed waiting for cluster="cluster-zstream-4-19-l2jj6j" in resourcegroup="rg-zstream-upgrade-4-19-hs9zt7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-l2jj6j in resource group rg-zstream-upgrade-4-19-hs9zt7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ece660>: 
    failed to create HCP cluster cluster-zstream-4-19-l2jj6j, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-l2jj6j in resource group rg-zstream-upgrade-4-19-hs9zt7, error: failed waiting for cluster="cluster-zstream-4-19-l2jj6j" in resourcegroup="rg-zstream-upgrade-4-19-hs9zt7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-l2jj6j in resource group rg-zstream-upgrade-4-19-hs9zt7, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012b33b0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-rdj6zw, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-rdj6zw in resource group rg-cp-ystream-upgrade-4-21-w9cgws, error: failed waiting for cluster="cp-ystream-upgrade-4-21-rdj6zw" in resourcegroup="rg-cp-ystream-upgrade-4-21-w9cgws" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-rdj6zw in resource group rg-cp-ystream-upgrade-4-21-w9cgws, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012b33b0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-rdj6zw, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-rdj6zw in resource group rg-cp-ystream-upgrade-4-21-w9cgws, error: failed waiting for cluster="cp-ystream-upgrade-4-21-rdj6zw" in resourcegroup="rg-cp-ystream-upgrade-4-21-w9cgws" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-rdj6zw in resource group rg-cp-ystream-upgrade-4-21-w9cgws, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc00150a420>: 
    failed to create HCP cluster cx-rg-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-pb9pq8, error: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-pb9pq8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-pb9pq8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc00150a420>: 
    failed to create HCP cluster cx-rg-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-pb9pq8, error: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-pb9pq8" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-pb9pq8, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009c6600>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-z5tdx7, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-z5tdx7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-z5tdx7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009c6600>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-z5tdx7, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-z5tdx7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-z5tdx7, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e00870>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-v87dq5, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-v87dq5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-v87dq5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e00870>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-v87dq5, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-v87dq5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-v87dq5, error: context deadline exceeded
    ...
occurred
MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equal
02:00:05 UTC · failed at e2e · test MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
    <string>: ...
to equal
    <string>: ...
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc001538870>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7xhvf7, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7xhvf7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7xhvf7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc001538870>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7xhvf7, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7xhvf7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7xhvf7, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a05800>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cqt74k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cqt74k, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a05800>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cqt74k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cqt74k, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:05 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ed6210>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qrdl2q, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qrdl2q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qrdl2q, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ed6210>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qrdl2q, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qrdl2q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qrdl2q, error: context deadline exceeded
    ...
occurred