No failure patterns match your search.
Environment: PROD
Failure patterns: 6 · Failures matched: 11 · Total runs: 6 · Runs affected: 6
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Interrupted by Userfull failure pattern: Interrupted by User Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 1 | 16.67%1 of 6 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsJun 4: 0 · Jun 5: 0 · Jun 6: 0 · Jun 7: 0 · Jun 8: 2 · Jun 9: 3 · Jun 10: 2 | none | |||||||||||||||||||||
Full failure examples (1)fail [:0]: Interrupted by User fail [:0]: Interrupted by User Contributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 3 | 50.00%3 of 6 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsJun 4: 0 · Jun 5: 0 · Jun 6: 0 · Jun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 3 | none | |||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-c7x88v/cluster-nightly-5-0-msgf8g should provision
Unexpected error:
<*fmt.wrapErrors | 0xc00053a0f0>:
failed waiting for cluster="cluster-nightly-5-0-msgf8g" in resourcegroup="rg-nightly-5-0-c7x88v" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-msgf8g in resource group rg-nightly-5-0-c7x88v, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-c7x88v/cluster-nightly-5-0-msgf8g should provision
Unexpected error:
<*fmt.wrapErrors | 0xc00053a0f0>:
failed waiting for cluster="cluster-nightly-5-0-msgf8g" in resourcegroup="rg-nightly-5-0-c7x88v" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-msgf8g in resource group rg-nightly-5-0-c7x88v, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-kcn82j/cluster-nightly-5-0-5r4lh5 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc001322450>:
failed waiting for cluster="cluster-nightly-5-0-5r4lh5" in resourcegroup="rg-nightly-5-0-kcn82j" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-5r4lh5 in resource group rg-nightly-5-0-kcn82j, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-kcn82j/cluster-nightly-5-0-5r4lh5 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc001322450>:
failed waiting for cluster="cluster-nightly-5-0-5r4lh5" in resourcegroup="rg-nightly-5-0-kcn82j" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-5r4lh5 in resource group rg-nightly-5-0-kcn82j, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-5xwj2c/cluster-nightly-5-0-9hzz55 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc0005304b0>:
failed waiting for cluster="cluster-nightly-5-0-9hzz55" in resourcegroup="rg-nightly-5-0-5xwj2c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-9hzz55 in resource group rg-nightly-5-0-5xwj2c, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-5xwj2c/cluster-nightly-5-0-9hzz55 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc0005304b0>:
failed waiting for cluster="cluster-nightly-5-0-9hzz55" in resourcegroup="rg-nightly-5-0-5xwj2c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-9hzz55 in resource group rg-nightly-5-0-5xwj2c, error: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 2 | 33.33%2 of 6 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsJun 4: 0 · Jun 5: 0 · Jun 6: 0 · Jun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 2 | none | |||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8"
Unexpected error:
<*fmt.wrapErrors | 0xc00083e120>:
failed to create NodePool np-jm8qc8, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-jm8qc8 in resource group rg-candidate-4-20-svkpkx, error: failed waiting for nodepool="np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8" in resourcegroup="rg-candidate-4-20-svkpkx" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8"
Unexpected error:
<*fmt.wrapErrors | 0xc00083e120>:
failed to create NodePool np-jm8qc8, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-jm8qc8 in resource group rg-candidate-4-20-svkpkx, error: failed waiting for nodepool="np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8" in resourcegroup="rg-candidate-4-20-svkpkx" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc00120e3f0>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-nlck2h, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-nlck2h" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc00120e3f0>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-nlck2h, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-nlck2h" to finish creating: context deadline exceeded
...
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | e2e | 1 | 16.67%1 of 6 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | Show trend detailsJun 4: 2 · Jun 5: 2 · Jun 6: 2 · Jun 7: 2 · Jun 8: 2 · Jun 9: 2 · Jun 10: 2 | none | |||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-9kjrzr at version 4.22.0
Unexpected error:
<*fmt.wrapError | 0xc000e08900>:
failed to create HCP cluster np-skip-minor-9kjrzr: failed starting cluster creation "np-skip-minor-9kjrzr" in resourcegroup="rg-np-skip-minor-9kjrzr-7gs6rf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9kjrzr-7gs6rf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-9kjrzr
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-9kjrzr at version 4.22.0
Unexpected error:
<*fmt.wrapError | 0xc000e08900>:
failed to create HCP cluster np-skip-minor-9kjrzr: failed starting cluster creation "np-skip-minor-9kjrzr" in resourcegroup="rg-np-skip-minor-9kjrzr-7gs6rf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9kjrzr-7gs6rf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-9kjrzr
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xc000b9caa0>:
failed to create HCP cluster np-dg-minor-9c7d7p: failed starting cluster creation "np-dg-minor-9c7d7p" in resourcegroup="rg-np-dg-minor-9c7d7p-rhs2s4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-9c7d7p-rhs2s4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-9c7d7p
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xc000b9caa0>:
failed to create HCP cluster np-dg-minor-9c7d7p: failed starting cluster creation "np-dg-minor-9c7d7p" in resourcegroup="rg-np-dg-minor-9c7d7p-rhs2s4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-9c7d7p-rhs2s4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-9c7d7p
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.19 to 4.19.33; provider Microsof...full failure pattern: ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.19 to 4.19.33; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 16.67%1 of 6 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJun 4: 0 · Jun 5: 0 · Jun 6: 0 · Jun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 1 | none | |||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
<*fmt.wrapError | 0xc000d0e180>:
failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-hf2g5m" resourcegroup="rg-np-dg-minor-hf2g5m-8ws4v8": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hf2g5m-8ws4v8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hf2g5m/nodePools/npdg-421-419
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
<*fmt.wrapError | 0xc000d0e180>:
failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-hf2g5m" resourcegroup="rg-np-dg-minor-hf2g5m-8ws4v8": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hf2g5m-8ws4v8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hf2g5m/nodePools/npdg-421-419
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.18": cannot downgrade from version 4.21.19 to 4.21.18; provider Microsof...full failure pattern: ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.18": cannot downgrade from version 4.21.19 to 4.21.18; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 16.67%1 of 6 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJun 4: 0 · Jun 5: 0 · Jun 6: 0 · Jun 7: 0 · Jun 8: 0 · Jun 9: 0 · Jun 10: 1 | none | |||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
<*fmt.wrapError | 0xc000dc8ea0>:
failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-62m6dt" resourcegroup="rg-np-downgrade-62m6dt-z875wp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-62m6dt-z875wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-62m6dt/nodePools/npdg-4-21
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
<*fmt.wrapError | 0xc000dc8ea0>:
failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-62m6dt" resourcegroup="rg-np-downgrade-62m6dt-z875wp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-62m6dt-z875wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-62m6dt/nodePools/npdg-4-21
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: MultipleErrorsOccurred
--------------------------------------------------------------------------------
{
"error": {
"code": "MultipleErrorsOccurred",
"message": "2 errors occurred",
"details": [
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
"target": "properties.version.id"
},
{
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
"target": "properties.version.id"
}
]
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||