No failure patterns match your search.
Environment: PROD
Failure patterns: 5 · Failures matched: 38 · Total runs: 1 · Runs affected: 1
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Deserializaion Error: no output from commandfull failure pattern: Deserializaion Error: no output from command Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 1 | 100.00%1 of 1 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsApr 22: 0 · Apr 23: 0 · Apr 24: 3 · Apr 25: 0 · Apr 26: 2 · Apr 27: 0 · Apr 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)ithub.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
> github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000e8eb88, {0xc000c39020?, 0x0?, 0x0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
> github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000d9f508, {0x0, 0x0}, {0xc284540, 0x0, 0x0}, {0xc00007c094, 0x2a}, 0x0, {0xc000c39000, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
> github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007c094, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d3ea85?, 0xc000a00000?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/ginkgo/util.go:85
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8da8ae8, 0xc000ac4aa0}, 0xc000ab3520, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:596
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:237
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:230
goroutine 118 [select]
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000ab3520?)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 117
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115 @ 04/28/26 04:31:29.581
"ts"="2026-04-28 04:31:29.581788" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-28 04:31:29.581897" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"="rg-candidate-4-22-fr8wfb"
"ts"="2026-04-28 04:31:29.754059" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"=""
"ts"="2026-04-28 04:31:30.945527" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
"ts"="2026-04-28 04:31:30.945809" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
"ts"="2026-04-28 04:31:30.945911" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-28 04:31:30.947534" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-120"
"ts"="2026-04-28 04:31:30.947616" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-22-fr8wfb"
"ts"="2026-04-28 04:31:31.115549" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="pooledNonMsiScopedAssignments-dwo7nh5uecnn2"
"ts"="2026-04-28 04:31:31.143526" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-28 04:31:31.256209" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn"
"ts"="2026-04-28 04:31:31.642441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="msi-deployment-120"
"ts"="2026-04-28 04:31:31.783617" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-28 04:31:32.120530" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-28 04:31:32.120780" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-22-fr8wfb"
Command Error: signal: killed
Deserializaion Error: no output from command
ithub.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
> github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000e8eb88, {0xc000c39020?, 0x0?, 0x0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
> github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000d9f508, {0x0, 0x0}, {0xc284540, 0x0, 0x0}, {0xc00007c094, 0x2a}, 0x0, {0xc000c39000, ...})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
> github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007c094, 0x2a}, ...)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2d3ea85?, 0xc000a00000?})
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/ginkgo/util.go:85
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8da8ae8, 0xc000ac4aa0}, 0xc000ab3520, 0x1)
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:596
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:237
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260408205138-ec501c2bf4a5/pkg/extension/extensiontests/spec.go:230
goroutine 118 [select]
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000ab3520?)
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
> github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 117
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
------------------------------
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115 @ 04/28/26 04:31:29.581
"ts"="2026-04-28 04:31:29.581788" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-04-28 04:31:29.581897" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"="rg-candidate-4-22-fr8wfb"
"ts"="2026-04-28 04:31:29.754059" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-candidate-4-22-47ghll-lz9frn" "resourceGroup"=""
"ts"="2026-04-28 04:31:30.945527" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
"ts"="2026-04-28 04:31:30.945809" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-53c251f3b5375d7cfda43a30036b1196777452fafe82c2917099d04531741969.yaml.gz"
"ts"="2026-04-28 04:31:30.945911" "level"=0 "msg"="collecting debug info"
"ts"="2026-04-28 04:31:30.947534" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-120"
"ts"="2026-04-28 04:31:30.947616" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-candidate-4-22-fr8wfb"
"ts"="2026-04-28 04:31:31.115549" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="pooledNonMsiScopedAssignments-dwo7nh5uecnn2"
"ts"="2026-04-28 04:31:31.143526" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-04-28 04:31:31.256209" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-candidate-4-22-fr8wfb" "deployment"="customer-infra-cluster-candidate-4-22-47ghll-lz9frn"
"ts"="2026-04-28 04:31:31.642441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-120" "deployment"="msi-deployment-120"
"ts"="2026-04-28 04:31:31.783617" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-04-28 04:31:32.120530" "level"=0 "msg"="deleting created resources"
"ts"="2026-04-28 04:31:32.120780" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-candidate-4-22-fr8wfb"
Command Error: signal: killed
Deserializaion Error: no output from commandContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster should become ready within 45 minutesfull failure pattern: Cluster should become ready within 45 minutes Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | e2e | 1 | 100.00%1 of 1 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | Show trend detailsApr 22: 0 · Apr 23: 1 · Apr 24: 1 · Apr 25: 1 · Apr 26: 1 · Apr 27: 0 · Apr 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
<bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
<bool>: ...
to be trueContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 1 | 100.00%1 of 1 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsApr 22: 0 · Apr 23: 0 · Apr 24: 0 · Apr 25: 0 · Apr 26: 0 · Apr 27: 0 · Apr 28: 34 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: Unexpected error:
<*fmt.wrapErrors | 0xc00113c630>:
failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-mk5cms" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: Unexpected error:
<*fmt.wrapErrors | 0xc00113c630>:
failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-mk5cms" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mk5cms, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115]: Unexpected error:
<*fmt.wrapErrors | 0xc00135bec0>:
failed to create HCP cluster cluster-candidate-4-20-4mvl7g, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: failed waiting for cluster="cluster-candidate-4-20-4mvl7g" in resourcegroup="rg-candidate-4-20-vm7tln" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:115]: Unexpected error:
<*fmt.wrapErrors | 0xc00135bec0>:
failed to create HCP cluster cluster-candidate-4-20-4mvl7g, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: failed waiting for cluster="cluster-candidate-4-20-4mvl7g" in resourcegroup="rg-candidate-4-20-vm7tln" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-candidate-4-20-4mvl7g in resource group rg-candidate-4-20-vm7tln, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapErrors | 0xc000f412f0>:
failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-c6ns4t" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapErrors | 0xc000f412f0>:
failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-c6ns4t" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-c6ns4t, error: context deadline exceeded
...
occurredContributing tests (34)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equalfull failure pattern: MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equal Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 1 | 100.00%1 of 1 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsApr 22: 0 · Apr 23: 3 · Apr 24: 3 · Apr 25: 1 · Apr 26: 0 · Apr 27: 0 · Apr 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
<string>: ...
to equal
<string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
<string>: ...
to equal
<string>: ...Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
context deadline exceededfull failure pattern: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 1 | 100.00%1 of 1 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsApr 22: 0 · Apr 23: 1 · Apr 24: 0 · Apr 25: 1 · Apr 26: 1 · Apr 27: 0 · Apr 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||