{"meta":{"start_date":"2026-10-25","end_date":"2026-10-25","start_at":"2026-10-25T00:00:00Z","end_at":"2026-10-26T00:00:00Z","timezone":"UTC","generated_at":"2026-10-30T07:25:19Z","environments":["prod"]},"environments":[{"environment":"prod","summary":{"total_runs":3,"failed_runs":2,"raw_occurrences":14,"matched_occurrences":14,"runs_affected":2},"rows":[{"environment":"prod","failure_pattern_id":"17960e7399b10596ca3db9acfee2784f8057d16e1a50930e2429e794d4813ec1","failure_pattern":"DNS for route host \u003croute-host\u003e did not resolve: DNS for \u003croute-host\u003e did not resolve within \u003cduration\u003e (last error: lookup \u003chost\u003e on \u003cdns-server\u003e: dial udp \u003cip\u003e: i/o timeout): context deadline exceeded","search_query":"DNS for route host agnhost-e2e-sample-app-4mmnj.apps.aro.h4m7c3s6a2p4j7a.3hkp.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4mmnj.apps.aro.h4m7c3s6a2p4j7a.3hkp.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4mmnj.apps.aro.h4m7c3s6a2p4j7a.3hkp.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded","failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20","occurrences":7,"runs_affected":1,"failed_runs":1,"run_impact_percent":33.333333333333336,"anchor_occurrences":7,"after_last_push_count":12,"trend_counts":[0,0,0,0,5,0,7],"trend_range":"2026-10-19..2026-10-25 UTC","prior_windows_present":0,"prior_runs_affected":3,"contributing_tests":[{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20","occurrences":1},{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21","occurrences":1},{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22","occurrences":1},{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0","occurrences":1},{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1","occurrences":1},{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create an HCP cluster with back-level version 4.20","occurrences":1},{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to retrieve cluster and nodepool status active versions","occurrences":1}],"full_error_samples":["fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster \"cluster-candidate-4-20-5gtk26\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0x187dce020660\u003e: \n    DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0x187dce1941c0\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster \"cluster-candidate-4-20-5gtk26\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0x187dce020660\u003e: \n    DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0x187dce1941c0\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }\noccurred","fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster \"v20261001\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0x33aec8dca6e0\u003e: \n    DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0x33aec9a24040\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster \"v20261001\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0x33aec8dca6e0\u003e: \n    DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0x33aec9a24040\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }\noccurred","fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20\nUnexpected error:\n    \u003c*fmt.wrapError | 0xaa72ff3eb40\u003e: \n    DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0xaa72fd7c9e0\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20\nUnexpected error:\n    \u003c*fmt.wrapError | 0xaa72ff3eb40\u003e: \n    DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0xaa72fd7c9e0\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }\noccurred"],"affected_runs":[{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"254849cbf5d8311752c99c9248a16fa04d7f6f2913e5fb83b7809a02a5a35916","pr_number":0,"after_last_push_of_merged_pr":true},{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"5eca3309ccd3e721e36f955090b7fbc956541d0a385657c2f9a6b84b63b4ed05","pr_number":0,"after_last_push_of_merged_pr":true},{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"63b9c0750fd00f7e0bfbdd70f47df6c7c8f5e5ec84bfe85b68318083b3ea2e5f","pr_number":0,"after_last_push_of_merged_pr":true},{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"7d2b6c735bdffa857d31b502d0f673ff158743f511cc94c645b379eec32fe280","pr_number":0,"after_last_push_of_merged_pr":true},{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"9d4340c3d5dd7a61b94734bf26b15b1bc36bf39a7633f96f82d30834ab5f4c8a","pr_number":0,"after_last_push_of_merged_pr":true},{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"daa48f2da7fe25cf04788befd5b4892e4a30494e022bdb921224e7051f37fd45","pr_number":0,"after_last_push_of_merged_pr":true},{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"e18d1395402ed17099a1459b75389ff956491b242d6bc7519d732c7e1751ed7c","pr_number":0,"after_last_push_of_merged_pr":true}]},{"environment":"prod","failure_pattern_id":"509ce29f121aa8889071be97ef0ea23faf908d0bfad41dc19a0232af98d72839","failure_pattern":"VerifySimpleWebApp failed: DNS for route host \u003croute-host\u003e did not resolve: DNS for \u003croute-host\u003e did not resolve within \u003cduration\u003e (last error: lookup \u003chost\u003e on \u003cdns-server\u003e: dial udp \u003cip\u003e: i/o timeout): context deadline exceeded","search_query":"VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded","failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create a HCP cluster and use cilium CNI plugin","occurrences":2,"runs_affected":1,"failed_runs":1,"run_impact_percent":33.333333333333336,"anchor_occurrences":2,"after_last_push_count":5,"trend_counts":[0,0,0,0,3,0,2],"trend_range":"2026-10-19..2026-10-25 UTC","prior_windows_present":0,"prior_runs_affected":2,"contributing_tests":[{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create a HCP cluster and use cilium CNI plugin","occurrences":1},{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration","occurrences":1}],"full_error_samples":["fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update\nUnexpected error:\n    \u003c*errors.joinError | 0x33a1f1315158\u003e: \n    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        errs: [\n            \u003c*fmt.wrapError | 0x33a1f0cea780\u003e{\n                msg: \"VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                err: \u003c*fmt.wrapError | 0x33a1f0bd2f00\u003e{\n                    msg: \"DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                    err: \u003c*fmt.wrapError | 0x33a1f13e4360\u003e{\n                        msg: \"DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                        err: \u003ccontext.deadlineExceededError\u003e{},\n                    },\n                },\n            },\n        ],\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update\nUnexpected error:\n    \u003c*errors.joinError | 0x33a1f1315158\u003e: \n    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        errs: [\n            \u003c*fmt.wrapError | 0x33a1f0cea780\u003e{\n                msg: \"VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                err: \u003c*fmt.wrapError | 0x33a1f0bd2f00\u003e{\n                    msg: \"DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                    err: \u003c*fmt.wrapError | 0x33a1f13e4360\u003e{\n                        msg: \"DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                        err: \u003ccontext.deadlineExceededError\u003e{},\n                    },\n                },\n            },\n        ],\n    }\noccurred","fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI\nUnexpected error:\n    \u003c*errors.joinError | 0xeeb7f5e0a68\u003e: \n    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        errs: [\n            \u003c*fmt.wrapError | 0xeeb7f3b69e0\u003e{\n                msg: \"VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                err: \u003c*fmt.wrapError | 0xeeb7f5137a0\u003e{\n                    msg: \"DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                    err: \u003c*fmt.wrapError | 0xeeb7f512040\u003e{\n                        msg: \"DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                        err: \u003ccontext.deadlineExceededError\u003e{},\n                    },\n                },\n            },\n        ],\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI\nUnexpected error:\n    \u003c*errors.joinError | 0xeeb7f5e0a68\u003e: \n    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\n    {\n        errs: [\n            \u003c*fmt.wrapError | 0xeeb7f3b69e0\u003e{\n                msg: \"VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                err: \u003c*fmt.wrapError | 0xeeb7f5137a0\u003e{\n                    msg: \"DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                    err: \u003c*fmt.wrapError | 0xeeb7f512040\u003e{\n                        msg: \"DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded\",\n                        err: \u003ccontext.deadlineExceededError\u003e{},\n                    },\n                },\n            },\n        ],\n    }\noccurred"],"affected_runs":[{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"0ac27c2c2596a79cd4371dcfd3d01134184bca542ebdf146502aaa026ae80229","pr_number":0,"after_last_push_of_merged_pr":true},{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"78fd5d6ed20f79e2553486e2f00d359fec4b08aeeed6379beabf7bf312ae9a76","pr_number":0,"after_last_push_of_merged_pr":true}]},{"environment":"prod","failure_pattern_id":"13f4d3d4d6df90cd306313783d31231d8428475383f7424332fdcb9914b386a1","failure_pattern":"DNS for route host agnhost-e2e-sample-app-\u003cid\u003e","search_query":"DNS for route host agnhost-e2e-sample-app-52rjm","failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should serve the default ingress certificate for 5.1","occurrences":1,"runs_affected":1,"failed_runs":1,"run_impact_percent":33.333333333333336,"anchor_occurrences":1,"after_last_push_count":1,"trend_counts":[0,0,0,0,0,0,1],"trend_range":"2026-10-19..2026-10-25 UTC","prior_windows_present":0,"prior_runs_affected":3,"contributing_tests":[{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"ARO-HCP should serve the default ingress certificate for 5.1","occurrences":1}],"full_error_samples":["fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate\nExpected success, but got an error:\n    \u003c*fmt.wrapError | 0x1d8dba328160\u003e: \n    DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0x1d8dba3280e0\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }\nfail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate\nExpected success, but got an error:\n    \u003c*fmt.wrapError | 0x1d8dba328160\u003e: \n    DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded\n    {\n        msg: \"DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded\",\n        err: \u003c*fmt.wrapError | 0x1d8dba3280e0\u003e{\n            msg: \"DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded\",\n            err: \u003ccontext.deadlineExceededError\u003e{},\n        },\n    }"],"affected_runs":[{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"076cacfa7561e03f7335026846eac17d1438353985ba609f1c98efdf0e3de954","pr_number":0,"after_last_push_of_merged_pr":true}]},{"environment":"prod","failure_pattern_id":"ef96fcb3cedbbf63865b36d0e47bdd70f7edc69989eb675136e8b13a4ecb570d","failure_pattern":"ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: DeadlineExceeded","failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits","occurrences":1,"runs_affected":1,"failed_runs":1,"run_impact_percent":33.333333333333336,"anchor_occurrences":1,"after_last_push_count":1,"trend_counts":[0,0,0,0,0,0,1],"trend_range":"2026-10-19..2026-10-25 UTC","prior_windows_present":0,"prior_runs_affected":1,"contributing_tests":[{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits","occurrences":1}],"full_error_samples":["fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0x22a837c45580\u003e: \n    failed to create NodePool autoscale-az: failed waiting for nodepool=\"autoscale-az\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m8kxkqwbf5w7\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: DeadlineExceeded\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940\",\n      \"name\": \"64b6150d-1b43-49ba-9a27-51fef85bb940\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-10-25T09:39:04.403310399Z\",\n      \"endTime\": \"2026-10-25T09:58:04.042996716Z\",\n      \"error\": {\n        \"code\": \"DeadlineExceeded\",\n        \"message\": \"node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    \u003cAzure error internals omitted\u003e\n        Azure request IDs:\n          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940\n            correlation ID: a329644d-c403-4d74-b2b5-fcc949c9882c\n            client request ID: unavailable\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0x22a837c45580\u003e: \n    failed to create NodePool autoscale-az: failed waiting for nodepool=\"autoscale-az\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m8kxkqwbf5w7\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: DeadlineExceeded\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940\",\n      \"name\": \"64b6150d-1b43-49ba-9a27-51fef85bb940\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-10-25T09:39:04.403310399Z\",\n      \"endTime\": \"2026-10-25T09:58:04.042996716Z\",\n      \"error\": {\n        \"code\": \"DeadlineExceeded\",\n        \"message\": \"node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    \u003cAzure error internals omitted\u003e\n        Azure request IDs:\n          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940\n            correlation ID: a329644d-c403-4d74-b2b5-fcc949c9882c\n            client request ID: unavailable\noccurred"],"affected_runs":[{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"eb4c9f692ebf3a9ebdd322c6273f89d4ee6b231c71d079f6aa6d3f3c0db70247","pr_number":0,"after_last_push_of_merged_pr":true}]},{"environment":"prod","failure_pattern_id":"fee9bdd5d83c8738b8bf37e9ed6800aab39015cbc2ff9fc672a130098b9fd967","failure_pattern":"VerifyBasicAccess failed: failed to list services: Unauthorized","search_query":"VerifyBasicAccess failed: failed to list services: Unauthorized","failed_at":"e2e","job_name":"periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly","test_name":"Customer should be able to use workload identity via the cluster OIDC issuer URL","occurrences":1,"runs_affected":1,"failed_runs":1,"run_impact_percent":33.333333333333336,"anchor_occurrences":1,"after_last_push_count":12,"trend_counts":[2,0,0,0,1,2,1],"trend_range":"2026-10-19..2026-10-25 UTC","prior_windows_present":2,"prior_window_starts":["2026-09-28","2026-10-05"],"prior_runs_affected":8,"prior_last_seen_at":"2026-10-11T23:00:04Z","contributing_tests":[{"failed_at":"e2e","job_name":"periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly","test_name":"Customer should be able to use workload identity via the cluster OIDC issuer URL","occurrences":1}],"full_error_samples":["fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable\nUnexpected error:\n    \u003c*errors.joinError | 0x5c10dd80030\u003e: \n    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized\n    VerifyBasicAccess failed: failed to list services: Unauthorized\n    {\n        errs: [\n            \u003c*fmt.wrapError | 0x5c10d35ddc0\u003e{\n                msg: \"VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized\",\n                err: \u003c*fmt.wrapError | 0x5c10d35dda0\u003e{\n                    msg: \"failed to list all APIServices: Unauthorized\",\n                    err: \u003c*errors.StatusError | 0x5c10db35540\u003e{\n                        ErrStatus: {\n                            TypeMeta: {Kind: \"\", APIVersion: \"\"},\n                            ListMeta: {\n                                SelfLink: \"\",\n                                ResourceVersion: \"\",\n                                Continue: \"\",\n                                RemainingItemCount: nil,\n                                ShardInfo: nil,\n                            },\n                            Status: \"Failure\",\n                            Message: \"Unauthorized\",\n                            Reason: \"Unauthorized\",\n                            Details: nil,\n                            Code: 401,\n                        },\n                    },\n                },\n            },\n            \u003c*fmt.wrapError | 0x5c10dd88020\u003e{\n                msg: \"VerifyBasicAccess failed: failed to list services: Unauthorized\",\n                err: \u003c*fmt.wrapError | 0x5c10dd88000\u003e{\n                    msg: \"failed to list services: Unauthorized\",\n                    err: \u003c*errors.StatusError | 0x5c10dbae320\u003e{\n                        ErrStatus: {\n                            TypeMeta: {Kind: \"\", APIVersion: \"\"},\n                            ListMeta: {\n                                SelfLink: \"\",\n                                ResourceVersion: \"\",\n                                Continue: \"\",\n                                RemainingItemCount: nil,\n                                ShardInfo: nil,\n                            },\n                            Status: \"Failure\",\n                            Message: \"Unauthorized\",\n                            Reason: \"Unauthorized\",\n                            Details: nil,\n                            Code: 401,\n                        },\n                    },\n                },\n            },\n        ],\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable\nUnexpected error:\n    \u003c*errors.joinError | 0x5c10dd80030\u003e: \n    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized\n    VerifyBasicAccess failed: failed to list services: Unauthorized\n    {\n        errs: [\n            \u003c*fmt.wrapError | 0x5c10d35ddc0\u003e{\n                msg: \"VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized\",\n                err: \u003c*fmt.wrapError | 0x5c10d35dda0\u003e{\n                    msg: \"failed to list all APIServices: Unauthorized\",\n                    err: \u003c*errors.StatusError | 0x5c10db35540\u003e{\n                        ErrStatus: {\n                            TypeMeta: {Kind: \"\", APIVersion: \"\"},\n                            ListMeta: {\n                                SelfLink: \"\",\n                                ResourceVersion: \"\",\n                                Continue: \"\",\n                                RemainingItemCount: nil,\n                                ShardInfo: nil,\n                            },\n                            Status: \"Failure\",\n                            Message: \"Unauthorized\",\n                            Reason: \"Unauthorized\",\n                            Details: nil,\n                            Code: 401,\n                        },\n                    },\n                },\n            },\n            \u003c*fmt.wrapError | 0x5c10dd88020\u003e{\n                msg: \"VerifyBasicAccess failed: failed to list services: Unauthorized\",\n                err: \u003c*fmt.wrapError | 0x5c10dd88000\u003e{\n                    msg: \"failed to list services: Unauthorized\",\n                    err: \u003c*errors.StatusError | 0x5c10dbae320\u003e{\n                        ErrStatus: {\n                            TypeMeta: {Kind: \"\", APIVersion: \"\"},\n                            ListMeta: {\n                                SelfLink: \"\",\n                                ResourceVersion: \"\",\n                                Continue: \"\",\n                                RemainingItemCount: nil,\n                                ShardInfo: nil,\n                            },\n                            Status: \"Failure\",\n                            Message: \"Unauthorized\",\n                            Reason: \"Unauthorized\",\n                            Details: nil,\n                            Code: 401,\n                        },\n                    },\n                },\n            },\n        ],\n    }\noccurred"],"affected_runs":[{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly/2114492215458795520","occurred_at":"2026-10-25T23:00:04Z","signature_id":"ead8c420db783411710bfdc18c3ac594eab0616d6007cfdac94e624578b57afc","pr_number":0,"after_last_push_of_merged_pr":true}]},{"environment":"prod","failure_pattern_id":"6f26d2f1140adc9b13b39b836a7ca29a860f5c4cb3811309f672f97a11466f4d","failure_pattern":"cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True","search_query":"cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)","failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create a cluster with an external auth config and get the external auth config","occurrences":1,"runs_affected":1,"failed_runs":1,"run_impact_percent":33.333333333333336,"anchor_occurrences":1,"after_last_push_count":2,"trend_counts":[0,0,0,0,1,0,1],"trend_range":"2026-10-19..2026-10-25 UTC","prior_windows_present":0,"prior_runs_affected":2,"contributing_tests":[{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should be able to create a cluster with an external auth config and get the external auth config","occurrences":1}],"full_error_samples":["fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation\nUnexpected error:\n    \u003c*errors.joinError | 0x471790ed8f0\u003e: \n    context deadline exceeded\n    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)\n    {\n        errs: [\n            \u003ccontext.deadlineExceededError\u003e{},\n            \u003c*errors.errorString | 0x4717a706200\u003e{\n                s: \"cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)\",\n            },\n        ],\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation\nUnexpected error:\n    \u003c*errors.joinError | 0x471790ed8f0\u003e: \n    context deadline exceeded\n    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)\n    {\n        errs: [\n            \u003ccontext.deadlineExceededError\u003e{},\n            \u003c*errors.errorString | 0x4717a706200\u003e{\n                s: \"cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)\",\n            },\n        ],\n    }\noccurred"],"affected_runs":[{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"928b1de9065772d3f2c7e201fb9ae101d2fb2df8e5b2cbd4ff7e82df961c73ea","pr_number":0,"after_last_push_of_merged_pr":true}]},{"environment":"prod","failure_pattern_id":"3ce2ba89b8e9686083a83dadac7501168f4efe9a7eacff619946e50e9db1c762","failure_pattern":"public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get \"\u003curl\u003e\": dial tcp: lookup \u003chost\u003e on \u003cdns-server\u003e: no such host","search_query":"public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get \"https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host","failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public","occurrences":1,"runs_affected":1,"failed_runs":1,"run_impact_percent":33.333333333333336,"anchor_occurrences":1,"after_last_push_count":1,"trend_counts":[0,0,0,0,0,0,1],"trend_range":"2026-10-19..2026-10-25 UTC","prior_windows_present":0,"prior_runs_affected":2,"contributing_tests":[{"failed_at":"e2e","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel","test_name":"Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public","occurrences":1}],"full_error_samples":["fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.\npublic ingress should be reachable from outside the VNet despite private KAS\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:\npublic ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get \"https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host\nUnexpected error:\n    \u003c*url.Error | 0x1bbed2a5bb90\u003e: \n    Get \"https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host\n    {\n        Op: \"Get\",\n        URL: \"https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\",\n        Err: \u003c*net.OpError | 0x1bbed27f1680\u003e{\n            Op: \"dial\",\n            Net: \"tcp\",\n            Source: nil,\n            Addr: nil,\n            Err: \u003c*net.DNSError | 0x1bbed27f1450\u003e{\n                UnwrapErr: nil,\n                Err: \"no such host\",\n                Name: \"console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\",\n                Server: \"172.30.0.10:53\",\n                IsTimeout: false,\n                IsTemporary: false,\n                IsNotFound: true,\n            },\n        },\n    }\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.\npublic ingress should be reachable from outside the VNet despite private KAS\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:\npublic ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get \"https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host\nUnexpected error:\n    \u003c*url.Error | 0x1bbed2a5bb90\u003e: \n    Get \"https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host\n    {\n        Op: \"Get\",\n        URL: \"https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\",\n        Err: \u003c*net.OpError | 0x1bbed27f1680\u003e{\n            Op: \"dial\",\n            Net: \"tcp\",\n            Source: nil,\n            Addr: nil,\n            Err: \u003c*net.DNSError | 0x1bbed27f1450\u003e{\n                UnwrapErr: nil,\n                Err: \"no such host\",\n                Name: \"console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io\",\n                Server: \"172.30.0.10:53\",\n                IsTimeout: false,\n                IsTemporary: false,\n                IsNotFound: true,\n            },\n        },\n    }\noccurred"],"affected_runs":[{"run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel/2114271800467656704","occurred_at":"2026-10-25T08:24:13Z","signature_id":"2e48272f7c6962293e37e27585cc8a5c9247db1c6e77cd891e766919f7108086","pr_number":0,"after_last_push_of_merged_pr":true}]}]}]}