No failure patterns match your search.
Environment: STG
Failure patterns: 4 · Failures matched: 11 · Total runs: 2 · Runs affected: 2
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment — New failure pattern — no prior history | e2e | 1 | 50.00%1 of 2 job runs affected | NoiseSignal: Noise — struct/object fragment — New failure pattern — no prior history | Show trend detailsOct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 2 | none | ||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-w7wgsb"
Expected success, but got an error:
<*fmt.wrapError | 0x80bfcd600c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x80bfcd60080>{
observation: <*errors.errorString | 0x80bfe902e80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x80bfcd60060>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-w7wgsb"
Expected success, but got an error:
<*fmt.wrapError | 0x80bfcd600c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x80bfcd60080>{
observation: <*errors.errorString | 0x80bfe902e80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x80bfcd60060>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-45dcr7"
Expected success, but got an error:
<*fmt.wrapError | 0x3387fb166020>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x3387fcff27f0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-45dcr7"
Expected success, but got an error:
<*fmt.wrapError | 0x3387fb166020>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x3387fcff27f0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 1 | 50.00%1 of 2 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 16: 1 · Oct 17: 2 · Oct 18: 1 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 5 | none | ||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x229a54bdd480>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x229a54bdd480>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x17090dbc6f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x17090dbc6f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2d643fc3b540>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2d643fc3b540>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (5)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 50.00%1 of 2 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 3 | none | ||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-7xkpjr"
Expected success, but got an error:
<*fmt.wrapError | 0x2e23e0966000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x2e23e2446680>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-7xkpjr"
Expected success, but got an error:
<*fmt.wrapError | 0x2e23e0966000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x2e23e2446680>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-gl2xgj"
Expected success, but got an error:
<*fmt.wrapError | 0x23574bbbc2a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x23574bbbc280>{
observation: <*errors.errorString | 0x23574d69e5e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x23574bbbc260>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-gl2xgj"
Expected success, but got an error:
<*fmt.wrapError | 0x23574bbbc2a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x23574bbbc280>{
observation: <*errors.errorString | 0x23574d69e5e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x23574bbbc260>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-20-5cm5db"
Expected success, but got an error:
<*fmt.wrapError | 0x86ee5d9c3c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x86ee5d9c3a0>{
observation: <*errors.errorString | 0x86ee53e2d80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x86ee5d9c380>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-20-5cm5db"
Expected success, but got an error:
<*fmt.wrapError | 0x86ee5d9c3c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x86ee5d9c3a0>{
observation: <*errors.errorString | 0x86ee53e2d80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x86ee5d9c380>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (3)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 50.00%1 of 2 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 | none | ||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x3cf97cd86ae0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-f4qstxjd2dn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233",
"name": "1ac5a43d-188f-41a4-873c-672951930233",
"status": "Failed",
"startTime": "2026-10-22T02:02:55.710362817Z",
"endTime": "2026-10-22T02:32:02.510983362Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
correlation ID: 9e15eb20-b5a4-4836-b8fc-ad2c0fd7a951
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x3cf97cd86ae0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-f4qstxjd2dn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233",
"name": "1ac5a43d-188f-41a4-873c-672951930233",
"status": "Failed",
"startTime": "2026-10-22T02:02:55.710362817Z",
"endTime": "2026-10-22T02:32:02.510983362Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
correlation ID: 9e15eb20-b5a4-4836-b8fc-ad2c0fd7a951
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||