<
Single Day: Jun 9
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 2
e2e (2)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:00:04 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001040b40>: 
    failed to create HCP cluster np-dg-minor-95jzkb: failed starting cluster creation "np-dg-minor-95jzkb" in resourcegroup="rg-np-dg-minor-95jzkb-g9rpcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-95jzkb-g9rpcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-95jzkb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001040b40>: 
    failed to create HCP cluster np-dg-minor-95jzkb: failed starting cluster creation "np-dg-minor-95jzkb" in resourcegroup="rg-np-dg-minor-95jzkb-g9rpcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-95jzkb-g9rpcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-95jzkb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:00:04 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-f684mv at version 4.22.0
Unexpected error:
    <*fmt.wrapError | 0xc000d63120>: 
    failed to create HCP cluster np-skip-minor-f684mv: failed starting cluster creation "np-skip-minor-f684mv" in resourcegroup="rg-np-skip-minor-f684mv-hgkqrz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-f684mv-hgkqrz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-f684mv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-f684mv at version 4.22.0
Unexpected error:
    <*fmt.wrapError | 0xc000d63120>: 
    failed to create HCP cluster np-skip-minor-f684mv: failed starting cluster creation "np-skip-minor-f684mv" in resourcegroup="rg-np-skip-minor-f684mv-hgkqrz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-f684mv-hgkqrz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-f684mv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
Interrupted by User
19:39:47 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 2
e2e (2)
Interrupted by User
19:29:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Interrupted by User
19:29:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
19:11:25 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-vmsnfk/cluster-nightly-5-0-6q4lgj should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0008e6870>: 
    failed waiting for cluster="cluster-nightly-5-0-6q4lgj" in resourcegroup="rg-nightly-5-0-vmsnfk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-6q4lgj in resource group rg-nightly-5-0-vmsnfk, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-vmsnfk/cluster-nightly-5-0-6q4lgj should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0008e6870>: 
    failed waiting for cluster="cluster-nightly-5-0-6q4lgj" in resourcegroup="rg-nightly-5-0-vmsnfk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-6q4lgj in resource group rg-nightly-5-0-vmsnfk, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 2
e2e (2)
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:18:46 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-c2nxww/cluster-nightly-5-0-vhm589 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00150c4e0>: 
    failed waiting for cluster="cluster-nightly-5-0-vhm589" in resourcegroup="rg-nightly-5-0-c2nxww" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-vhm589 in resource group rg-nightly-5-0-c2nxww, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-c2nxww/cluster-nightly-5-0-vhm589 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00150c4e0>: 
    failed waiting for cluster="cluster-nightly-5-0-vhm589" in resourcegroup="rg-nightly-5-0-c2nxww" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-vhm589 in resource group rg-nightly-5-0-c2nxww, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:18:46 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-vcfqbx/cluster-nightly-4-23-fnbphq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001268690>: 
    failed waiting for cluster="cluster-nightly-4-23-fnbphq" in resourcegroup="rg-nightly-4-23-vcfqbx" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-fnbphq in resource group rg-nightly-4-23-vcfqbx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-vcfqbx/cluster-nightly-4-23-fnbphq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001268690>: 
    failed waiting for cluster="cluster-nightly-4-23-fnbphq" in resourcegroup="rg-nightly-4-23-vcfqbx" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-fnbphq in resource group rg-nightly-4-23-vcfqbx, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 5
e2e (5)
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:16:51 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapErrors | 0xc0009485a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-kkjnrm" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-kkjnrm, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapErrors | 0xc0009485a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-kkjnrm" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-kkjnrm, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:16:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-9s2nzg/cluster-nightly-4-23-5jn2lq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001370930>: 
    failed waiting for cluster="cluster-nightly-4-23-5jn2lq" in resourcegroup="rg-nightly-4-23-9s2nzg" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-5jn2lq in resource group rg-nightly-4-23-9s2nzg, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-9s2nzg/cluster-nightly-4-23-5jn2lq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001370930>: 
    failed waiting for cluster="cluster-nightly-4-23-5jn2lq" in resourcegroup="rg-nightly-4-23-9s2nzg" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-5jn2lq in resource group rg-nightly-4-23-9s2nzg, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:16:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-szhbft/cluster-nightly-5-0-t5fcjl should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001165c20>: 
    failed waiting for cluster="cluster-nightly-5-0-t5fcjl" in resourcegroup="rg-nightly-5-0-szhbft" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-t5fcjl in resource group rg-nightly-5-0-szhbft, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-szhbft/cluster-nightly-5-0-t5fcjl should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001165c20>: 
    failed waiting for cluster="cluster-nightly-5-0-t5fcjl" in resourcegroup="rg-nightly-5-0-szhbft" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-t5fcjl in resource group rg-nightly-5-0-szhbft, error: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
17:16:51 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0009906f0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-852q76, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-852q76" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0009906f0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-852q76, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-852q76" to finish creating: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
17:16:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-ggxw2n" for cluster "cluster-candidate-4-21-ggxw2n"
Unexpected error:
    <*fmt.wrapErrors | 0xc0005d87b0>: 
    failed to create NodePool np-ggxw2n, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-ggxw2n in resource group rg-candidate-4-21-chmwt8, error: failed waiting for nodepool="np-ggxw2n" for cluster "cluster-candidate-4-21-ggxw2n" in resourcegroup="rg-candidate-4-21-chmwt8" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-ggxw2n" for cluster "cluster-candidate-4-21-ggxw2n"
Unexpected error:
    <*fmt.wrapErrors | 0xc0005d87b0>: 
    failed to create NodePool np-ggxw2n, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-ggxw2n in resource group rg-candidate-4-21-chmwt8, error: failed waiting for nodepool="np-ggxw2n" for cluster "cluster-candidate-4-21-ggxw2n" in resourcegroup="rg-candidate-4-21-chmwt8" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 9
e2e (9)
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:37:22 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc00109eb40>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-675sc9, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-675sc9" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-675sc9, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc00109eb40>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-675sc9, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-675sc9" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-675sc9, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:37:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc001602600>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2cm842" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2cm842, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc001602600>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2cm842" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2cm842, error: context deadline exceeded
    ...
occurred
Cluster should become ready within '20.000000' minutes
11:37:22 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:37:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-mx2rw2/cluster-nightly-4-23-clnb9r should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001074ab0>: 
    failed waiting for cluster="cluster-nightly-4-23-clnb9r" in resourcegroup="rg-nightly-4-23-mx2rw2" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-clnb9r in resource group rg-nightly-4-23-mx2rw2, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-mx2rw2/cluster-nightly-4-23-clnb9r should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001074ab0>: 
    failed waiting for cluster="cluster-nightly-4-23-clnb9r" in resourcegroup="rg-nightly-4-23-mx2rw2" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-clnb9r in resource group rg-nightly-4-23-mx2rw2, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:37:22 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapErrors | 0xc000f01710>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-pp74jp, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-pp74jp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-pp74jp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapErrors | 0xc000f01710>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-pp74jp, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-pp74jp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-pp74jp, error: context deadline exceeded
    ...
occurred
cluster has not yet reached Succeeded state
11:37:22 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:37:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-vf9czb/cluster-nightly-5-0-jk5fg7 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc000fc96b0>: 
    failed waiting for cluster="cluster-nightly-5-0-jk5fg7" in resourcegroup="rg-nightly-5-0-vf9czb" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-jk5fg7 in resource group rg-nightly-5-0-vf9czb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-vf9czb/cluster-nightly-5-0-jk5fg7 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc000fc96b0>: 
    failed waiting for cluster="cluster-nightly-5-0-jk5fg7" in resourcegroup="rg-nightly-5-0-vf9czb" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-jk5fg7 in resource group rg-nightly-5-0-vf9czb, error: context deadline exceeded
    ...
occurred
tls: failed to verify certificate: x509: certificate signed by unknown authority
11:37:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:205]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-qrwxz9"
Unexpected error:
    <*fmt.wrapError | 0xc000997620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-92ldj.apps.aro.r8i7j7z8f3c5v9h.00gq.eastus2euap.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:205]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-qrwxz9"
Unexpected error:
    <*fmt.wrapError | 0xc000997620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-92ldj.apps.aro.r8i7j7z8f3c5v9h.00gq.eastus2euap.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
11:37:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-hzgwbr" for cluster "cluster-candidate-4-21-hzgwbr"
Unexpected error:
    <*fmt.wrapErrors | 0xc001293e30>: 
    failed to create NodePool np-hzgwbr, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-hzgwbr in resource group rg-candidate-4-21-9d9sd9, error: failed waiting for nodepool="np-hzgwbr" for cluster "cluster-candidate-4-21-hzgwbr" in resourcegroup="rg-candidate-4-21-9d9sd9" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-hzgwbr" for cluster "cluster-candidate-4-21-hzgwbr"
Unexpected error:
    <*fmt.wrapErrors | 0xc001293e30>: 
    failed to create NodePool np-hzgwbr, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-hzgwbr in resource group rg-candidate-4-21-9d9sd9, error: failed waiting for nodepool="np-hzgwbr" for cluster "cluster-candidate-4-21-hzgwbr" in resourcegroup="rg-candidate-4-21-9d9sd9" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:34:04 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapErrors | 0xc00132ee10>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-2s6t8f, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2s6t8f" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-2s6t8f, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapErrors | 0xc00132ee10>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-2s6t8f, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-2s6t8f" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-2s6t8f, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:32:59 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapErrors | 0xc0004fc690>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-r7ldlr, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-r7ldlr" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-r7ldlr, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapErrors | 0xc0004fc690>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-r7ldlr, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-r7ldlr" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-r7ldlr, error: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 12
e2e (12)
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc0014de450>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-5whn7c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-5whn7c, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc0014de450>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-5whn7c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-5whn7c, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-20-cqlcfn" with version 4.20.23 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc001900c90>: 
    failed to create HCP cluster cluster-zstream-4-20-cqlcfn, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-cqlcfn in resource group rg-zstream-upgrade-4-20-ft2rr4, error: failed waiting for cluster="cluster-zstream-4-20-cqlcfn" in resourcegroup="rg-zstream-upgrade-4-20-ft2rr4" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-cqlcfn in resource group rg-zstream-upgrade-4-20-ft2rr4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-20-cqlcfn" with version 4.20.23 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc001900c90>: 
    failed to create HCP cluster cluster-zstream-4-20-cqlcfn, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-cqlcfn in resource group rg-zstream-upgrade-4-20-ft2rr4, error: failed waiting for cluster="cluster-zstream-4-20-cqlcfn" in resourcegroup="rg-zstream-upgrade-4-20-ft2rr4" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-cqlcfn in resource group rg-zstream-upgrade-4-20-ft2rr4, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapErrors | 0xc00159a6c0>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rs9vtl, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rs9vtl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rs9vtl, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapErrors | 0xc00159a6c0>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rs9vtl, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rs9vtl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rs9vtl, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137]: HCP cluster rg-candidate-4-21-7g8djb/cluster-candidate-4-21-xmtssq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0000eae10>: 
    failed to create HCP cluster cluster-candidate-4-21-xmtssq, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-xmtssq in resource group rg-candidate-4-21-7g8djb, error: failed waiting for cluster="cluster-candidate-4-21-xmtssq" in resourcegroup="rg-candidate-4-21-7g8djb" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-xmtssq in resource group rg-candidate-4-21-7g8djb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137]: HCP cluster rg-candidate-4-21-7g8djb/cluster-candidate-4-21-xmtssq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0000eae10>: 
    failed to create HCP cluster cluster-candidate-4-21-xmtssq, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-xmtssq in resource group rg-candidate-4-21-7g8djb, error: failed waiting for cluster="cluster-candidate-4-21-xmtssq" in resourcegroup="rg-candidate-4-21-7g8djb" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-xmtssq in resource group rg-candidate-4-21-7g8djb, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-b5kj4q/cluster-nightly-4-23-c6nrfd should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0010aa330>: 
    failed waiting for cluster="cluster-nightly-4-23-c6nrfd" in resourcegroup="rg-nightly-4-23-b5kj4q" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-c6nrfd in resource group rg-nightly-4-23-b5kj4q, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-b5kj4q/cluster-nightly-4-23-c6nrfd should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0010aa330>: 
    failed waiting for cluster="cluster-nightly-4-23-c6nrfd" in resourcegroup="rg-nightly-4-23-b5kj4q" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-c6nrfd in resource group rg-nightly-4-23-b5kj4q, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137]: HCP cluster rg-candidate-4-22-bhks5n/cluster-candidate-4-22-np4fcf should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00180f320>: 
    failed to create HCP cluster cluster-candidate-4-22-np4fcf, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-np4fcf in resource group rg-candidate-4-22-bhks5n, error: failed waiting for cluster="cluster-candidate-4-22-np4fcf" in resourcegroup="rg-candidate-4-22-bhks5n" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-np4fcf in resource group rg-candidate-4-22-bhks5n, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137]: HCP cluster rg-candidate-4-22-bhks5n/cluster-candidate-4-22-np4fcf should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00180f320>: 
    failed to create HCP cluster cluster-candidate-4-22-np4fcf, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-np4fcf in resource group rg-candidate-4-22-bhks5n, error: failed waiting for cluster="cluster-candidate-4-22-np4fcf" in resourcegroup="rg-candidate-4-22-bhks5n" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-np4fcf in resource group rg-candidate-4-22-bhks5n, error: context deadline exceeded
    ...
occurred
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.18 to 4.19.33; provider Microsoft.RedHatOpenShift
02:00:03 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc0002b8420>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-bzsg8d" resourcegroup="rg-np-dg-minor-bzsg8d-gwv8l6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-bzsg8d-gwv8l6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-bzsg8d/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.18 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc0002b8420>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-bzsg8d" resourcegroup="rg-np-dg-minor-bzsg8d-gwv8l6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-bzsg8d-gwv8l6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-bzsg8d/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.18 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-22-phvfxj" with version 4.22.0-rc.4 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000d1af00>: 
    failed to create HCP cluster cluster-zstream-4-22-phvfxj, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-phvfxj in resource group rg-zstream-upgrade-4-22-g9lt82, error: failed waiting for cluster="cluster-zstream-4-22-phvfxj" in resourcegroup="rg-zstream-upgrade-4-22-g9lt82" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-phvfxj in resource group rg-zstream-upgrade-4-22-g9lt82, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-22-phvfxj" with version 4.22.0-rc.4 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000d1af00>: 
    failed to create HCP cluster cluster-zstream-4-22-phvfxj, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-phvfxj in resource group rg-zstream-upgrade-4-22-g9lt82, error: failed waiting for cluster="cluster-zstream-4-22-phvfxj" in resourcegroup="rg-zstream-upgrade-4-22-g9lt82" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-phvfxj in resource group rg-zstream-upgrade-4-22-g9lt82, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-19-dv6mdg" with version 4.19.31 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000bd9ad0>: 
    failed to create HCP cluster cluster-zstream-4-19-dv6mdg, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-dv6mdg in resource group rg-zstream-upgrade-4-19-brhmk9, error: failed waiting for cluster="cluster-zstream-4-19-dv6mdg" in resourcegroup="rg-zstream-upgrade-4-19-brhmk9" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-dv6mdg in resource group rg-zstream-upgrade-4-19-brhmk9, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-19-dv6mdg" with version 4.19.31 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000bd9ad0>: 
    failed to create HCP cluster cluster-zstream-4-19-dv6mdg, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-dv6mdg in resource group rg-zstream-upgrade-4-19-brhmk9, error: failed waiting for cluster="cluster-zstream-4-19-dv6mdg" in resourcegroup="rg-zstream-upgrade-4-19-brhmk9" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-dv6mdg in resource group rg-zstream-upgrade-4-19-brhmk9, error: context deadline exceeded
    ...
occurred
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.17": cannot downgrade from version 4.21.18 to 4.21.17; provider Microsoft.RedHatOpenShift
02:00:03 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.17
Unexpected error:
    <*fmt.wrapError | 0xc000a18200>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-9hkgrv" resourcegroup="rg-np-downgrade-9hkgrv-4rg65k": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-9hkgrv-4rg65k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-9hkgrv/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": invalid node pool version 4.21.17: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": cannot downgrade from version 4.21.18 to 4.21.17",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.17
Unexpected error:
    <*fmt.wrapError | 0xc000a18200>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-9hkgrv" resourcegroup="rg-np-downgrade-9hkgrv-4rg65k": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-9hkgrv-4rg65k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-9hkgrv/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": invalid node pool version 4.21.17: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": cannot downgrade from version 4.21.18 to 4.21.17",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapErrors | 0xc000af0900>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-brgdg7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-brgdg7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapErrors | 0xc000af0900>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-brgdg7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-brgdg7, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:03 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-21-2bd9v7" with version 4.21.17 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00173e930>: 
    failed to create HCP cluster cluster-zstream-4-21-2bd9v7, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2bd9v7 in resource group rg-zstream-upgrade-4-21-g2d2z5, error: failed waiting for cluster="cluster-zstream-4-21-2bd9v7" in resourcegroup="rg-zstream-upgrade-4-21-g2d2z5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2bd9v7 in resource group rg-zstream-upgrade-4-21-g2d2z5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:137]: failed to create HCP cluster "cluster-zstream-4-21-2bd9v7" with version 4.21.17 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00173e930>: 
    failed to create HCP cluster cluster-zstream-4-21-2bd9v7, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2bd9v7 in resource group rg-zstream-upgrade-4-21-g2d2z5, error: failed waiting for cluster="cluster-zstream-4-21-2bd9v7" in resourcegroup="rg-zstream-upgrade-4-21-g2d2z5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2bd9v7 in resource group rg-zstream-upgrade-4-21-g2d2z5, error: context deadline exceeded
    ...
occurred