<
Weekly: Jun 8 - Jun 14
> Reset
View JSON API

Environment: PROD

Failure patterns: 17 · Failures matched: 261 · Total runs: 44 · Runs affected: 38

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
e2e2352.27%23 of 44 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsJun 7: 0 · Jun 8: 44 · Jun 9: 23 · Jun 10: 3 · Jun 11: 5 · Jun 12: 82 · Jun 13: 4 · Jun 14: 8none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapErrors | 0xc001512570>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-tn428m, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-tn428m" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-tn428m, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapErrors | 0xc001512570>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-tn428m, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-tn428m" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-tn428m, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-v2vrn5/cluster-nightly-4-23-qz5lb5 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001756060>: 
    failed waiting for cluster="cluster-nightly-4-23-qz5lb5" in resourcegroup="rg-nightly-4-23-v2vrn5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-qz5lb5 in resource group rg-nightly-4-23-v2vrn5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-v2vrn5/cluster-nightly-4-23-qz5lb5 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001756060>: 
    failed waiting for cluster="cluster-nightly-4-23-qz5lb5" in resourcegroup="rg-nightly-4-23-v2vrn5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-4-23-qz5lb5 in resource group rg-nightly-4-23-v2vrn5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137]: HCP cluster rg-nightly-4-20-tvmzpn/cluster-nightly-4-20-rp6bj2 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc000eb12c0>: 
    failed to create HCP cluster cluster-nightly-4-20-rp6bj2, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-20-rp6bj2 in resource group rg-nightly-4-20-tvmzpn, error: failed waiting for cluster="cluster-nightly-4-20-rp6bj2" in resourcegroup="rg-nightly-4-20-tvmzpn" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-20-rp6bj2 in resource group rg-nightly-4-20-tvmzpn, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137]: HCP cluster rg-nightly-4-20-tvmzpn/cluster-nightly-4-20-rp6bj2 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc000eb12c0>: 
    failed to create HCP cluster cluster-nightly-4-20-rp6bj2, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-20-rp6bj2 in resource group rg-nightly-4-20-tvmzpn, error: failed waiting for cluster="cluster-nightly-4-20-rp6bj2" in resourcegroup="rg-nightly-4-20-tvmzpn" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-20-rp6bj2 in resource group rg-nightly-4-20-tvmzpn, error: context deadline exceeded
    ...
occurred
Contributing tests (61)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.011
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.238
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates6
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.234
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.234
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.04
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.223
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.193
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.203
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelMISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.193
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.203
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.213
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.223
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelMISE Routing routes to the correct frontend based on version header default route returns no version header2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should update NodeDrainTimeoutMinutes and upgrade a nodepool1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.191
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (23)
Date (UTC)Associated PRProw job
n/a2066294654927638528
n/a2065977561912971264
n/a2065932263165530112
n/a2065615170868088832
n/a2065569872280031232
n/a2065511780783755264
n/a2065388059075022848
n/a2065252825906024448
n/a2065207457541525504
n/a2065133461022183424
n/a2064999166156738560
n/a2064790437436067840
n/a2064600993957416960
n/a2064601088287313920
n/a2064425150199107584
n/a2064396799321313280
n/a2064396319065116672
n/a2064310885882531840
n/a2064279856266350592
n/a2064264482309279744
n/a2064165596991328256
n/a2064061230653378560
n/a2064015992266166272
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
e2e1329.55%13 of 44 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsJun 7: 0 · Jun 8: 0 · Jun 9: 3 · Jun 10: 2 · Jun 11: 4 · Jun 12: 10 · Jun 13: 8 · Jun 14: 8none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-w5pt4m" for cluster "cluster-nightly-4-21-w5pt4m"
Unexpected error:
    <*fmt.wrapErrors | 0xc000976450>: 
    failed to create NodePool np-w5pt4m, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-w5pt4m in resource group rg-nightly-4-21-x7zgfd, error: failed waiting for nodepool="np-w5pt4m" for cluster "cluster-nightly-4-21-w5pt4m" in resourcegroup="rg-nightly-4-21-x7zgfd" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-w5pt4m" for cluster "cluster-nightly-4-21-w5pt4m"
Unexpected error:
    <*fmt.wrapErrors | 0xc000976450>: 
    failed to create NodePool np-w5pt4m, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-w5pt4m in resource group rg-nightly-4-21-x7zgfd, error: failed waiting for nodepool="np-w5pt4m" for cluster "cluster-nightly-4-21-w5pt4m" in resourcegroup="rg-nightly-4-21-x7zgfd" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-t2zvsb" for cluster "cluster-nightly-4-22-t2zvsb"
Unexpected error:
    <*fmt.wrapErrors | 0xc000c15050>: 
    failed to create NodePool np-t2zvsb, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-t2zvsb in resource group rg-nightly-4-22-fsszt4, error: failed waiting for nodepool="np-t2zvsb" for cluster "cluster-nightly-4-22-t2zvsb" in resourcegroup="rg-nightly-4-22-fsszt4" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-t2zvsb" for cluster "cluster-nightly-4-22-t2zvsb"
Unexpected error:
    <*fmt.wrapErrors | 0xc000c15050>: 
    failed to create NodePool np-t2zvsb, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-t2zvsb in resource group rg-nightly-4-22-fsszt4, error: failed waiting for nodepool="np-t2zvsb" for cluster "cluster-nightly-4-22-t2zvsb" in resourcegroup="rg-nightly-4-22-fsszt4" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0013d4438>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-5nhhnb, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-5nhhnb" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0013d4438>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-5nhhnb, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-5nhhnb" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (14)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.215
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.204
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.213
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.223
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (13)
Date (UTC)Associated PRProw job
n/a2066294654927638528
n/a2065977561912971264
n/a2065932263165530112
n/a2065615170868088832
n/a2065569872280031232
n/a2065303808690360320
n/a2064999166156738560
n/a2064892795172163584
n/a2064892701127479296
n/a2064600993957416960
n/a2064601088287313920
n/a2064396319065116672
n/a2064310885882531840
Interrupted by User
full failure pattern:
Interrupted by User
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e1022.73%10 of 44 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsJun 7: 0 · Jun 8: 2 · Jun 9: 3 · Jun 10: 2 · Jun 11: 4 · Jun 12: 5 · Jun 13: 0 · Jun 14: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:30]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:30]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:207]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:207]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:46]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:46]: Interrupted by User
Contributing tests (10)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.204
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
Affected runs (10)
Date (UTC)Associated PRProw job
n/a2065511780783755264
n/a2065388059075022848
n/a2064995434417360896
n/a2064979974896488448
n/a2064892701127479296
n/a2064892559024459776
n/a2064599912179306496
n/a2064444081047605248
n/a2064436508076544000
n/a2064061230653378560
tls: failed to verify certificate: x509: certificate signed by unknown authority
full failure pattern:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e24.55%2 of 44 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsJun 7: 1 · Jun 8: 1 · Jun 9: 1 · Jun 10: 0 · Jun 11: 0 · Jun 12: 0 · Jun 13: 0 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:205]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-qrwxz9"
Unexpected error:
    <*fmt.wrapError | 0xc000997620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-92ldj.apps.aro.r8i7j7z8f3c5v9h.00gq.eastus2euap.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:205]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-qrwxz9"
Unexpected error:
    <*fmt.wrapError | 0xc000997620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-92ldj.apps.aro.r8i7j7z8f3c5v9h.00gq.eastus2euap.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:205]: failed to verify simple web app runs on cluster "cluster-nightly-5-0-49vmww"
Unexpected error:
    <*fmt.wrapError | 0xc0000d5080>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-h7tkh.apps.aro.g5x8x9n2r7b0d4z.mce0.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:205]: failed to verify simple web app runs on cluster "cluster-nightly-5-0-49vmww"
Unexpected error:
    <*fmt.wrapError | 0xc0000d5080>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-h7tkh.apps.aro.g5x8x9n2r7b0d4z.mce0.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2064310885882531840
n/a2064120297979645952
Deserializaion Error: no output from command
full failure pattern:
Deserializaion Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error
e2e36.82%3 of 44 job runs affectedNoiseSignal: Noise — source deserialization/no-output errorShow trend detailsJun 7: 0 · Jun 8: 1 · Jun 9: 0 · Jun 10: 0 · Jun 11: 0 · Jun 12: 3 · Jun 13: 0 · Jun 14: 0none
Full failure examples (3)
077e8 sp=0xc000e077e0 pc=0x2ed3cc1
created by os/signal.Notify.func1.1 in goroutine 84
	/usr/lib/golang/src/os/signal/signal.go:152 +0x1f

goroutine 104 gp=0xc000201180 m=nil [runnable]:
runtime.gopark(0xc000e07f88?, 0x3?, 0x0?, 0x0?, 0xc000e07f6a?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000e07df0 sp=0xc000e07dd0 pc=0x2ecb50e
runtime.selectgo(0xc000e07f88, 0xc000e07f64, 0x0?, 0x0, 0xc000000000?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000e07f30 sp=0xc000e07df0 pc=0x2ea62f7
github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000efd1e0?)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131 +0x7d fp=0xc000e07fc8 sp=0xc000e07f30 pc=0x33174bd
github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.gowrap1()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:158 +0x24 fp=0xc000e07fe0 sp=0xc000e07fc8 pc=0x3317404
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e07fe8 sp=0xc000e07fe0 pc=0x2ed3cc1
created by github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 30
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128 +0x165

goroutine 30 gp=0xc0002016c0 m=nil [select, 149 minutes]:
runtime.gopark(0xc000fa4990?, 0x5?, 0x28?, 0x39?, 0xc000fa3cee?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000fa38f0 sp=0xc000fa38d0 pc=0x2ecb50e
runtime.selectgo(0xc000fa4990, 0xc000fa3ce4, 0x0?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000fa3a30 sp=0xc000fa38f0 pc=0x2ea62f7
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode(_, {0x4b, 0x4, {0x861dc94, 0x8}, 0xc000e0fb80, {{0xa91353c, 0x5b}, 0xd0, {0x0, ...}, ...}, ...}, ...)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:923 +0xf65 fp=0xc000fa4d28 sp=0xc000fa3a30 pc=0x33422c5
github.com/onsi/ginkgo/v2/internal.(*group).attemptSpec(0xc000fa6b88, 0x1, {{0xc000ec8e08?, 0xc00070c8c0?, 0x2?}, 0x0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199 +0xbd3 fp=0xc000fa5d48 sp=0xc000fa4d28 pc=0x3327713
github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000fa6b88, {0xc000a9e500?, 0x0?, 0x0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349 +0x905 fp=0xc000fa6950 sp=0xc000fa5d48 pc=0x3329ca5
github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000a4e008, {0x0, 0x0}, {0xc81b8c0, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000a9e4e0, ...})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496 +0xa9b fp=0xc000fa6e00 sp=0xc000fa6950 pc=0x333d79b
github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68 +0x3c5 fp=0xc000fa6ea8 sp=0xc000fa6e00 pc=0x3344b05
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2e5aa85?, 0xc0002b0240?})
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/ginkgo/util.go:84 +0x2ba fp=0xc000fa7b78 sp=0xc000fa6ea8 pc=0x6f4bb5a
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x91e1708, 0xc000e98ff0}, 0xc000efd1e0, 0x1)
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:596 +0xee fp=0xc000fa7c20 sp=0xc000fa7b78 pc=0x6f41dce
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:237 +0x196 fp=0xc000fa7fe0 sp=0xc000fa7c20 pc=0x6f3db36
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000fa7fe8 sp=0xc000fa7fe0 pc=0x2ed3cc1
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:230 +0x1e5

rax    0xca
rbx    0x0
rcx    0x2ed5ae3
rdx    0x0
rdi    0xc7f0b70
rsi    0x80
rbp    0x7fffd43d8bf0
rsp    0x7fffd43d8ba8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1
r13    0x1
r14    0xc7ee5e0
r15    0x5
rip    0x2ed5ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
077e8 sp=0xc000e077e0 pc=0x2ed3cc1
created by os/signal.Notify.func1.1 in goroutine 84
	/usr/lib/golang/src/os/signal/signal.go:152 +0x1f

goroutine 104 gp=0xc000201180 m=nil [runnable]:
runtime.gopark(0xc000e07f88?, 0x3?, 0x0?, 0x0?, 0xc000e07f6a?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000e07df0 sp=0xc000e07dd0 pc=0x2ecb50e
runtime.selectgo(0xc000e07f88, 0xc000e07f64, 0x0?, 0x0, 0xc000000000?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000e07f30 sp=0xc000e07df0 pc=0x2ea62f7
github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000efd1e0?)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131 +0x7d fp=0xc000e07fc8 sp=0xc000e07f30 pc=0x33174bd
github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.gowrap1()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:158 +0x24 fp=0xc000e07fe0 sp=0xc000e07fc8 pc=0x3317404
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e07fe8 sp=0xc000e07fe0 pc=0x2ed3cc1
created by github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 30
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128 +0x165

goroutine 30 gp=0xc0002016c0 m=nil [select, 149 minutes]:
runtime.gopark(0xc000fa4990?, 0x5?, 0x28?, 0x39?, 0xc000fa3cee?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000fa38f0 sp=0xc000fa38d0 pc=0x2ecb50e
runtime.selectgo(0xc000fa4990, 0xc000fa3ce4, 0x0?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000fa3a30 sp=0xc000fa38f0 pc=0x2ea62f7
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode(_, {0x4b, 0x4, {0x861dc94, 0x8}, 0xc000e0fb80, {{0xa91353c, 0x5b}, 0xd0, {0x0, ...}, ...}, ...}, ...)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:923 +0xf65 fp=0xc000fa4d28 sp=0xc000fa3a30 pc=0x33422c5
github.com/onsi/ginkgo/v2/internal.(*group).attemptSpec(0xc000fa6b88, 0x1, {{0xc000ec8e08?, 0xc00070c8c0?, 0x2?}, 0x0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199 +0xbd3 fp=0xc000fa5d48 sp=0xc000fa4d28 pc=0x3327713
github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000fa6b88, {0xc000a9e500?, 0x0?, 0x0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349 +0x905 fp=0xc000fa6950 sp=0xc000fa5d48 pc=0x3329ca5
github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000a4e008, {0x0, 0x0}, {0xc81b8c0, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000a9e4e0, ...})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496 +0xa9b fp=0xc000fa6e00 sp=0xc000fa6950 pc=0x333d79b
github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68 +0x3c5 fp=0xc000fa6ea8 sp=0xc000fa6e00 pc=0x3344b05
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2e5aa85?, 0xc0002b0240?})
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/ginkgo/util.go:84 +0x2ba fp=0xc000fa7b78 sp=0xc000fa6ea8 pc=0x6f4bb5a
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x91e1708, 0xc000e98ff0}, 0xc000efd1e0, 0x1)
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:596 +0xee fp=0xc000fa7c20 sp=0xc000fa7b78 pc=0x6f41dce
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:237 +0x196 fp=0xc000fa7fe0 sp=0xc000fa7c20 pc=0x6f3db36
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000fa7fe8 sp=0xc000fa7fe0 pc=0x2ed3cc1
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:230 +0x1e5

rax    0xca
rbx    0x0
rcx    0x2ed5ae3
rdx    0x0
rdi    0xc7f0b70
rsi    0x80
rbp    0x7fffd43d8bf0
rsp    0x7fffd43d8ba8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1
r13    0x1
r14    0xc7ee5e0
r15    0x5
rip    0x2ed5ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
{0xc00007a094, 0x2a}, 0x0, {0xc0007868e0, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2e5aa85?, 0xc000ab0d80?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/ginkgo/util.go:84
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x91e1708, 0xc000e157c0}, 0xc000a57380, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:230

          goroutine 88 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000a57380?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 87
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137 @ 06/12/26 13:49:03.006
  "ts"="2026-06-12 13:49:03.007583" "level"=0 "msg"="===== TEST CASE ENDED: FAILURE ====="
  "ts"="2026-06-12 13:49:03.007634" "level"=0 "msg"="===== CLEANUP BEGAN ====="
  "ts"="2026-06-12 13:49:03.007761" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-06-12 13:49:03.007887" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-20-5pvqrv-67ffr4" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 13:49:03.075105" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-20-5pvqrv-67ffr4" "resourceGroup"=""
  "ts"="2026-06-12 13:49:03.778877" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d4ee2620a775558818d65bdad60edec7ff8775b05730402bc0de88c0d1c3d86f.yaml.gz"
  "ts"="2026-06-12 13:49:03.779022" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d4ee2620a775558818d65bdad60edec7ff8775b05730402bc0de88c0d1c3d86f.yaml.gz"
  "ts"="2026-06-12 13:49:03.779140" "level"=0 "msg"="collecting debug info"
  "ts"="2026-06-12 13:49:03.781368" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 13:49:03.781396" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-148"
  "ts"="2026-06-12 13:49:04.152510" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-20-vpbqs7" "deployment"="pooledNonMsiScopedAssignments-34bborbknryhu"
  "ts"="2026-06-12 13:49:04.166599" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-06-12 13:49:04.243967" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-20-vpbqs7" "deployment"="customer-infra-cluster-candidate-4-20-5pvqrv-67ffr4"
  "ts"="2026-06-12 13:49:04.475079" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="msi-deployment-148"
  "ts"="2026-06-12 13:49:04.599107" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-06-12 13:49:05.015191" "level"=0 "msg"="deleting created resources"
  "ts"="2026-06-12 13:49:05.015430" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 14:02:43.842163" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 14:02:43.842189" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-candidate-4-20-vpbqs7"
Command Error: signal: killed
Deserializaion Error: no output from command
{0xc00007a094, 0x2a}, 0x0, {0xc0007868e0, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2e5aa85?, 0xc000ab0d80?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/ginkgo/util.go:84
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x91e1708, 0xc000e157c0}, 0xc000a57380, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260422120742-50dec0b5ecf3/pkg/extension/extensiontests/spec.go:230

          goroutine 88 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000a57380?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 87
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:137 @ 06/12/26 13:49:03.006
  "ts"="2026-06-12 13:49:03.007583" "level"=0 "msg"="===== TEST CASE ENDED: FAILURE ====="
  "ts"="2026-06-12 13:49:03.007634" "level"=0 "msg"="===== CLEANUP BEGAN ====="
  "ts"="2026-06-12 13:49:03.007761" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-06-12 13:49:03.007887" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-20-5pvqrv-67ffr4" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 13:49:03.075105" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-20-5pvqrv-67ffr4" "resourceGroup"=""
  "ts"="2026-06-12 13:49:03.778877" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d4ee2620a775558818d65bdad60edec7ff8775b05730402bc0de88c0d1c3d86f.yaml.gz"
  "ts"="2026-06-12 13:49:03.779022" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d4ee2620a775558818d65bdad60edec7ff8775b05730402bc0de88c0d1c3d86f.yaml.gz"
  "ts"="2026-06-12 13:49:03.779140" "level"=0 "msg"="collecting debug info"
  "ts"="2026-06-12 13:49:03.781368" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 13:49:03.781396" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-148"
  "ts"="2026-06-12 13:49:04.152510" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-20-vpbqs7" "deployment"="pooledNonMsiScopedAssignments-34bborbknryhu"
  "ts"="2026-06-12 13:49:04.166599" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-06-12 13:49:04.243967" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-20-vpbqs7" "deployment"="customer-infra-cluster-candidate-4-20-5pvqrv-67ffr4"
  "ts"="2026-06-12 13:49:04.475079" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="msi-deployment-148"
  "ts"="2026-06-12 13:49:04.599107" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-06-12 13:49:05.015191" "level"=0 "msg"="deleting created resources"
  "ts"="2026-06-12 13:49:05.015430" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 14:02:43.842163" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-candidate-4-20-vpbqs7"
  "ts"="2026-06-12 14:02:43.842189" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-candidate-4-20-vpbqs7"
Command Error: signal: killed
Deserializaion Error: no output from command
000aa5188, {0x914ca80, 0xc000125120}, 0xc001904a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0019049e8 sp=0xc0019049b0 pc=0x319c0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000aa5188, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001904c60 sp=0xc0019049e8 pc=0x319917b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000aa5188, {0xc0006af000, 0x1000, 0xc000308ba0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001904cc8 sp=0xc001904c60 pc=0x319fa85
net/http.(*persistConn).Read(0xc000aff320, {0xc0006af000?, 0x3289b05?, 0x9145b00?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001904d28 sp=0xc001904cc8 pc=0x3291407
bufio.(*Reader).fill(0xc000f00480)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001904d60 sp=0xc001904d28 pc=0x3031943
bufio.(*Reader).Peek(0xc000f00480, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001904d80 sp=0xc001904d60 pc=0x3031ad3
net/http.(*persistConn).readLoop(0xc000aff320)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001904fc8 sp=0xc001904d80 pc=0x3291f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001904fe0 sp=0xc001904fc8 pc=0x32908a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001904fe8 sp=0xc001904fe0 pc=0x2ed3cc1
created by net/http.(*Transport).dialConn in goroutine 129
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 203 gp=0xc001600700 m=nil [sync.WaitGroup.Wait, 4 minutes]:
runtime.gopark(0xc0015fb5b0?, 0x2e62874?, 0x80?, 0x24?, 0x200c0013e0bd0?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0010f9550 sp=0xc0010f9530 pc=0x2ecb50e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc0013ea778, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc0010f95b8 sp=0xc0010f9550 pc=0x2ea759d
sync.runtime_SemacquireWaitGroup(0xc000519e20?, 0x0?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc0010f95f0 sp=0xc0010f95b8 pc=0x2eccc2e
sync.(*WaitGroup).Wait(0xc0013ea770)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc0010f9618 sp=0xc0010f95f0 pc=0x2ef0405
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000b99180, {0x7fa434753b08, 0xc00159b860}, {{0xc000ab4fb0, 0x1, 0x1}, 0x34630b8a000, {0x861de1c, 0x8}, {{0x0, ...}, ...}})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc0010f9730 sp=0xc0010f9618 pc=0x6bae3d2
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000b99180, {0x7fa434753b08, 0xc00159b860})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc0010f9888 sp=0xc0010f9730 pc=0x6badb30
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7fa434753b08?, 0xc00159b860?})
	<autogenerated>:1 +0x33 fp=0xc0010f98b0 sp=0xc0010f9888 pc=0x6bc9cd3
runtime.call16(0xc00177cc90, 0xc00069bf70, 0x0, 0x0, 0x0, 0x10, 0xc0010f9b08)
	/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc0010f98d0 sp=0xc0010f98b0 pc=0x2ed2223
runtime.reflectcall(0x7ff4600?, 0xc00159b860?, 0x2?, 0x865590b?, 0x0?, 0x12?, 0x7bbaac0?)
	<autogenerated>:1 +0x36 fp=0xc0010f9910 sp=0xc0010f98d0 pc=0x2edaab6
reflect.Value.call({0x746fa60?, 0xc00069bf70?, 0xc000e8f3c8?}, {0x86117f8, 0x4}, {0xc00190ff60, 0x1, 0xc000d1ed70?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc0010f9e70 sp=0xc0010f9910 pc=0x2f0dc66
reflect.Value.Call({0x746fa60?, 0xc00069bf70?, 0x86102f5?}, {0xc00190ff60?, 0xc000e8f3c8?, 0x0?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc0010f9ee8 sp=0xc0010f9e70 pc=0x2f0cd59
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x91fba30?, 0xc00159b860?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc0010f9f88 sp=0xc0010f9ee8 pc=0x3331170
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc0010f9fe0 sp=0xc0010f9f88 pc=0x3343c3b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0010f9fe8 sp=0xc0010f9fe0 pc=0x2ed3cc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 47
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x2ed5ae3
rdx    0x0
rdi    0xc7f2cb0
rsi    0x80
rbp    0x7fa43cb09cb8
rsp    0x7fa43cb09c70
r8     0x0
r9     0x0
r10    0x7fa43cb09ca8
r11    0x202
r12    0x7fa43cb09ca8
r13    0x1
r14    0xc000002c40
r15    0x0
rip    0x2ed5ae3
rflags 0x202
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
000aa5188, {0x914ca80, 0xc000125120}, 0xc001904a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0019049e8 sp=0xc0019049b0 pc=0x319c0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000aa5188, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001904c60 sp=0xc0019049e8 pc=0x319917b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000aa5188, {0xc0006af000, 0x1000, 0xc000308ba0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001904cc8 sp=0xc001904c60 pc=0x319fa85
net/http.(*persistConn).Read(0xc000aff320, {0xc0006af000?, 0x3289b05?, 0x9145b00?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001904d28 sp=0xc001904cc8 pc=0x3291407
bufio.(*Reader).fill(0xc000f00480)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001904d60 sp=0xc001904d28 pc=0x3031943
bufio.(*Reader).Peek(0xc000f00480, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001904d80 sp=0xc001904d60 pc=0x3031ad3
net/http.(*persistConn).readLoop(0xc000aff320)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001904fc8 sp=0xc001904d80 pc=0x3291f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001904fe0 sp=0xc001904fc8 pc=0x32908a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001904fe8 sp=0xc001904fe0 pc=0x2ed3cc1
created by net/http.(*Transport).dialConn in goroutine 129
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 203 gp=0xc001600700 m=nil [sync.WaitGroup.Wait, 4 minutes]:
runtime.gopark(0xc0015fb5b0?, 0x2e62874?, 0x80?, 0x24?, 0x200c0013e0bd0?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0010f9550 sp=0xc0010f9530 pc=0x2ecb50e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc0013ea778, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc0010f95b8 sp=0xc0010f9550 pc=0x2ea759d
sync.runtime_SemacquireWaitGroup(0xc000519e20?, 0x0?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc0010f95f0 sp=0xc0010f95b8 pc=0x2eccc2e
sync.(*WaitGroup).Wait(0xc0013ea770)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc0010f9618 sp=0xc0010f95f0 pc=0x2ef0405
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000b99180, {0x7fa434753b08, 0xc00159b860}, {{0xc000ab4fb0, 0x1, 0x1}, 0x34630b8a000, {0x861de1c, 0x8}, {{0x0, ...}, ...}})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc0010f9730 sp=0xc0010f9618 pc=0x6bae3d2
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000b99180, {0x7fa434753b08, 0xc00159b860})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc0010f9888 sp=0xc0010f9730 pc=0x6badb30
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7fa434753b08?, 0xc00159b860?})
	<autogenerated>:1 +0x33 fp=0xc0010f98b0 sp=0xc0010f9888 pc=0x6bc9cd3
runtime.call16(0xc00177cc90, 0xc00069bf70, 0x0, 0x0, 0x0, 0x10, 0xc0010f9b08)
	/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc0010f98d0 sp=0xc0010f98b0 pc=0x2ed2223
runtime.reflectcall(0x7ff4600?, 0xc00159b860?, 0x2?, 0x865590b?, 0x0?, 0x12?, 0x7bbaac0?)
	<autogenerated>:1 +0x36 fp=0xc0010f9910 sp=0xc0010f98d0 pc=0x2edaab6
reflect.Value.call({0x746fa60?, 0xc00069bf70?, 0xc000e8f3c8?}, {0x86117f8, 0x4}, {0xc00190ff60, 0x1, 0xc000d1ed70?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc0010f9e70 sp=0xc0010f9910 pc=0x2f0dc66
reflect.Value.Call({0x746fa60?, 0xc00069bf70?, 0x86102f5?}, {0xc00190ff60?, 0xc000e8f3c8?, 0x0?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc0010f9ee8 sp=0xc0010f9e70 pc=0x2f0cd59
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x91fba30?, 0xc00159b860?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc0010f9f88 sp=0xc0010f9ee8 pc=0x3331170
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc0010f9fe0 sp=0xc0010f9f88 pc=0x3343c3b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0010f9fe8 sp=0xc0010f9fe0 pc=0x2ed3cc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 47
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x2ed5ae3
rdx    0x0
rdi    0xc7f2cb0
rsi    0x80
rbp    0x7fa43cb09cb8
rsp    0x7fa43cb09c70
r8     0x0
r9     0x0
r10    0x7fa43cb09ca8
r11    0x202
r12    0x7fa43cb09ca8
r13    0x1
r14    0xc000002c40
r15    0x0
rip    0x2ed5ae3
rflags 0x202
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelMISE Routing routes to the correct frontend based on version header default route returns no version header1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2065511780783755264
n/a2065388059075022848
n/a2064061230653378560
ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Compute
full failure pattern:
ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e511.36%5 of 44 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsJun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 0 · Jun 11: 0 · Jun 12: 1 · Jun 13: 2 · Jun 14: 2none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc000d273a0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-9xbp9m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-9xbp9m/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is '21503528-9017-4784-8156-8f2140dfcfce'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc000d273a0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-9xbp9m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-9xbp9m/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is '21503528-9017-4784-8156-8f2140dfcfce'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc0000d5ae0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-7ktm25": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-7ktm25/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'df96e1a8-3336-4b6e-8a5f-4f6ea39c547e'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc0000d5ae0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-7ktm25": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-7ktm25/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'df96e1a8-3336-4b6e-8a5f-4f6ea39c547e'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc0010931e0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-fjsk59": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-fjsk59/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'af5eb3b1-d28d-40aa-b393-7c5df027bafd'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc0010931e0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-fjsk59": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-fjsk59/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'af5eb3b1-d28d-40aa-b393-7c5df027bafd'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2066294654927638528
n/a2065977561912971264
n/a2065932263165530112
n/a2065615170868088832
n/a2065569872280031232
Cluster should become ready within '20.000000' minutes
full failure pattern:
Cluster should become ready within '20.000000' minutes
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e49.09%4 of 44 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsJun 7: 0 · Jun 8: 1 · Jun 9: 1 · Jun 10: 0 · Jun 11: 0 · Jun 12: 2 · Jun 13: 0 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.006s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.006s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle4
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2065511780783755264
n/a2065388059075022848
n/a2064310885882531840
n/a2064061230653378560
cluster has not yet reached Succeeded state
full failure pattern:
cluster has not yet reached Succeeded state
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e49.09%4 of 44 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsJun 7: 0 · Jun 8: 1 · Jun 9: 1 · Jun 10: 0 · Jun 11: 0 · Jun 12: 2 · Jun 13: 0 · Jun 14: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.007s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.007s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation4
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2065511780783755264
n/a2065388059075022848
n/a2064310885882531840
n/a2064061230653378560
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e36.82%3 of 44 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsJun 7: 2 · Jun 8: 2 · Jun 9: 2 · Jun 10: 2 · Jun 11: 0 · Jun 12: 0 · Jun 13: 0 · Jun 14: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-9kjrzr at version 4.22.0
Unexpected error:
    <*fmt.wrapError | 0xc000e08900>: 
    failed to create HCP cluster np-skip-minor-9kjrzr: failed starting cluster creation "np-skip-minor-9kjrzr" in resourcegroup="rg-np-skip-minor-9kjrzr-7gs6rf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9kjrzr-7gs6rf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-9kjrzr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-9kjrzr at version 4.22.0
Unexpected error:
    <*fmt.wrapError | 0xc000e08900>: 
    failed to create HCP cluster np-skip-minor-9kjrzr: failed starting cluster creation "np-skip-minor-9kjrzr" in resourcegroup="rg-np-skip-minor-9kjrzr-7gs6rf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9kjrzr-7gs6rf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-9kjrzr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9caa0>: 
    failed to create HCP cluster np-dg-minor-9c7d7p: failed starting cluster creation "np-dg-minor-9c7d7p" in resourcegroup="rg-np-dg-minor-9c7d7p-rhs2s4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-9c7d7p-rhs2s4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-9c7d7p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9caa0>: 
    failed to create HCP cluster np-dg-minor-9c7d7p: failed starting cluster creation "np-dg-minor-9c7d7p" in resourcegroup="rg-np-dg-minor-9c7d7p-rhs2s4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-9c7d7p-rhs2s4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-9c7d7p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001040b40>: 
    failed to create HCP cluster np-dg-minor-95jzkb: failed starting cluster creation "np-dg-minor-95jzkb" in resourcegroup="rg-np-dg-minor-95jzkb-g9rpcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-95jzkb-g9rpcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-95jzkb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001040b40>: 
    failed to create HCP cluster np-dg-minor-95jzkb: failed starting cluster creation "np-dg-minor-95jzkb" in resourcegroup="rg-np-dg-minor-95jzkb-g9rpcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-95jzkb-g9rpcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-95jzkb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest3
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2064845065389346816
n/a2064482691524333568
n/a2064120297979645952
failed to wait for first cluster "<cluster>" to complete creation
full failure pattern:
failed to wait for first cluster "<cluster>" to complete creation
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e36.82%3 of 44 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsJun 7: 0 · Jun 8: 1 · Jun 9: 0 · Jun 10: 0 · Jun 11: 0 · Jun 12: 2 · Jun 13: 0 · Jun 14: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group3
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2065511780783755264
n/a2065388059075022848
n/a2064061230653378560
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.17": cannot downgrade from version 4.21.18 to 4.21.17; provider Microsof...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.17": cannot downgrade from version 4.21.18 to 4.21.17; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e24.55%2 of 44 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsJun 7: 1 · Jun 8: 1 · Jun 9: 1 · Jun 10: 0 · Jun 11: 0 · Jun 12: 0 · Jun 13: 0 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.17
Unexpected error:
    <*fmt.wrapError | 0xc000a18200>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-9hkgrv" resourcegroup="rg-np-downgrade-9hkgrv-4rg65k": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-9hkgrv-4rg65k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-9hkgrv/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": invalid node pool version 4.21.17: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": cannot downgrade from version 4.21.18 to 4.21.17",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.17
Unexpected error:
    <*fmt.wrapError | 0xc000a18200>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-9hkgrv" resourcegroup="rg-np-downgrade-9hkgrv-4rg65k": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-9hkgrv-4rg65k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-9hkgrv/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": invalid node pool version 4.21.17: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": cannot downgrade from version 4.21.18 to 4.21.17",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.17
Unexpected error:
    <*fmt.wrapError | 0xc000134780>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-z52pvc" resourcegroup="rg-np-downgrade-z52pvc-w5x8s7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-z52pvc-w5x8s7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-z52pvc/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": invalid node pool version 4.21.17: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": cannot downgrade from version 4.21.18 to 4.21.17",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.17
Unexpected error:
    <*fmt.wrapError | 0xc000134780>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-z52pvc" resourcegroup="rg-np-downgrade-z52pvc-w5x8s7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-z52pvc-w5x8s7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-z52pvc/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": invalid node pool version 4.21.17: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.17\": cannot downgrade from version 4.21.18 to 4.21.17",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2064165596991328256
n/a2063803203840905216
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.18 to 4.19.33; provider Microsof...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.18 to 4.19.33; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e24.55%2 of 44 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsJun 7: 1 · Jun 8: 1 · Jun 9: 1 · Jun 10: 0 · Jun 11: 0 · Jun 12: 0 · Jun 13: 0 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc0002b8420>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-bzsg8d" resourcegroup="rg-np-dg-minor-bzsg8d-gwv8l6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-bzsg8d-gwv8l6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-bzsg8d/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.18 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc0002b8420>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-bzsg8d" resourcegroup="rg-np-dg-minor-bzsg8d-gwv8l6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-bzsg8d-gwv8l6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-bzsg8d/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.18 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc00039e480>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-mkwzwg" resourcegroup="rg-np-dg-minor-mkwzwg-pcf6kp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-mkwzwg-pcf6kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-mkwzwg/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.18 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc00039e480>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-mkwzwg" resourcegroup="rg-np-dg-minor-mkwzwg-pcf6kp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-mkwzwg-pcf6kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-mkwzwg/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.18",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.18 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2064165596991328256
n/a2063803203840905216
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e24.55%2 of 44 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 0 · Jun 11: 0 · Jun 12: 0 · Jun 13: 2 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000df03f0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-bn486z, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bn486z" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000df03f0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-bn486z, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bn486z" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00181a420>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-vdrfcr, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-vdrfcr" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00181a420>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-vdrfcr, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-vdrfcr" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2065932263165530112
n/a2065615170868088832
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.19 to 4.19.33; provider Microsof...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.19 to 4.19.33; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e24.55%2 of 44 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 1 · Jun 11: 1 · Jun 12: 0 · Jun 13: 0 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc000f04b20>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-c9l7d4" resourcegroup="rg-np-dg-minor-c9l7d4-jlrlpp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-c9l7d4-jlrlpp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-c9l7d4/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc000f04b20>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-c9l7d4" resourcegroup="rg-np-dg-minor-c9l7d4-jlrlpp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-c9l7d4-jlrlpp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-c9l7d4/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc000d0e180>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-hf2g5m" resourcegroup="rg-np-dg-minor-hf2g5m-8ws4v8": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hf2g5m-8ws4v8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hf2g5m/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc000d0e180>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-hf2g5m" resourcegroup="rg-np-dg-minor-hf2g5m-8ws4v8": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hf2g5m-8ws4v8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hf2g5m/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2064890364723990528
n/a2064527988531138560
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.18": cannot downgrade from version 4.21.19 to 4.21.18; provider Microsof...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.18": cannot downgrade from version 4.21.19 to 4.21.18; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e24.55%2 of 44 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 1 · Jun 11: 1 · Jun 12: 0 · Jun 13: 0 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
    <*fmt.wrapError | 0xc00099a220>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-sh8sbm" resourcegroup="rg-np-downgrade-sh8sbm-8xc2g7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-sh8sbm-8xc2g7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-sh8sbm/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
    <*fmt.wrapError | 0xc00099a220>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-sh8sbm" resourcegroup="rg-np-downgrade-sh8sbm-8xc2g7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-sh8sbm-8xc2g7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-sh8sbm/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
    <*fmt.wrapError | 0xc000dc8ea0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-62m6dt" resourcegroup="rg-np-downgrade-62m6dt-z875wp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-62m6dt-z875wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-62m6dt/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
    <*fmt.wrapError | 0xc000dc8ea0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-62m6dt" resourcegroup="rg-np-downgrade-62m6dt-z875wp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-62m6dt-z875wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-62m6dt/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2064890364723990528
n/a2064527988531138560
</html>
full failure pattern:
</html>
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.27%1 of 44 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 0 · Jun 11: 0 · Jun 12: 2 · Jun 13: 0 · Jun 14: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:75]: failed to get latest install version for nightly channel: after 4 attempts: query nightly tags for 5.0.0-0.nightly-multi returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:75]: failed to get latest install version for nightly channel: after 4 attempts: query nightly tags for 5.0.0-0.nightly-multi returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:75]: failed to get latest install version for nightly channel: after 4 attempts: query nightly tags for 4.23.0-0.nightly-multi returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:75]: failed to get latest install version for nightly channel: after 4 attempts: query nightly tags for 4.23.0-0.nightly-multi returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2065303808690360320
Post "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
Post "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.27%1 of 44 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 0 · Jun 11: 1 · Jun 12: 0 · Jun 13: 0 · Jun 14: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 2
Unexpected error:
    <*url.Error | 0xc000999560>: 
    Post "https://api.n1i3e7e9x8g0b6s.ci57.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 2
Unexpected error:
    <*url.Error | 0xc000999560>: 
    Post "https://api.n1i3e7e9x8g0b6s.ci57.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2064995434417360896