{"meta":{"date":"2026-06-22","timezone":"UTC","generated_at":"2026-08-29T12:30:18Z","environments":["stg"]},"environments":[{"environment":"stg","summary":{"total_runs":6,"failed_runs":5,"runs_with_occurrences":5,"runs_with_failure_pattern_matches":5,"runs_with_unmatched_occurrences":0,"runs_without_occurrence_rows":1,"failed_runs_without_occurrence_rows":0},"runs":[{"run":{"environment":"stg","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly/2069193738747383808","job_name":"periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-06-22T23:00:05Z"},"failed_at":["e2e"],"failed_test_count":4,"occurrences":[{"row_id":"2d876c2f6e5f0a8f59c4528c216592ecb26a5cdf327b386bce9705349d430214","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly/2069193738747383808","occurred_at":"2026-06-22T23:00:05Z","failed_at":"e2e","signature_id":"f3fabf19c99a298b418328ac0114b6d63c61ee4e7af72157649bb72b8a5321b8","test_name":"ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:\nUnexpected error:\n    \u003c*errors.joinError | 0xc000012ba0\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"delayed-rbac-cluster\" in resourcegroup=\"delayed-rbac-kjlqnf\" to finish deleting: context canceled\n    ...\noccurred  \nfail [:0]: A node timeout occurred\nfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:\nUnexpected error:\n    \u003c*errors.joinError | 0xc000012ba0\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"delayed-rbac-cluster\" in resourcegroup=\"delayed-rbac-kjlqnf\" to finish deleting: context canceled\n    ...\noccurred  \nfail [:0]: A node timeout occurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7f0e44476112ce4e675f687843407c03482d80d265239de5b2ddbd45c441a04e","failure_pattern":"failed to cleanup resource group: at least one HCP cluster \u003ccluster\u003e to delete: failed waiting for hcpcluster=\"\u003ccluster\u003e\" in resourcegroup=\"\u003cresource-group\u003e\" to finish deleting: context canceled","search_query":"failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"delayed-rbac-cluster\" in resourcegroup=\"delayed-rbac-kjlqnf\" to finish deleting: context canceled"}},{"row_id":"53f0da01ceb55ea3035c351438c5ee6f4e8135c7508a78c1e7f30937c902fe79","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly/2069193738747383808","occurred_at":"2026-06-22T23:00:05Z","failed_at":"e2e","signature_id":"b499dfd9e3a9bdd7f99096eb91aa8936a5522c33190c6da69f6dd19013892363","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-22-6f2d8n/cluster-nightly-4-22-vbj6q7 should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0018126f0\u003e: \n    failed to create HCP cluster cluster-nightly-4-22-vbj6q7, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: failed waiting for cluster=\"cluster-nightly-4-22-vbj6q7\" in resourcegroup=\"rg-nightly-4-22-6f2d8n\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-22-6f2d8n/cluster-nightly-4-22-vbj6q7 should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0018126f0\u003e: \n    failed to create HCP cluster cluster-nightly-4-22-vbj6q7, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: failed waiting for cluster=\"cluster-nightly-4-22-vbj6q7\" in resourcegroup=\"rg-nightly-4-22-6f2d8n\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster=\"cluster-candidate-4-20-rb4p4z\" in resourcegroup=\"rg-candidate-4-20-8n2xlf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded"}},{"row_id":"67d001ab730108eec6d3093a2739cd8a18e1bc288e7a8eeff47daa7dc8c3423f","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly/2069193738747383808","occurred_at":"2026-06-22T23:00:05Z","failed_at":"e2e","signature_id":"cf0298768bf3d36c59220c7a0cfb551998ec62877185c8fc3712300cbfb4c581","test_name":"Customer should be able to lifecycle and confirm external auth on a cluster","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:\nUnexpected error:\n    \u003c*errors.joinError | 0xc001800348\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listzr65l9\" in resourcegroup=\"ea-list-z9h2z8\" to finish deleting: context canceled\n    ...\noccurred  \nfail [:0]: A node timeout occurred\nfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:\nUnexpected error:\n    \u003c*errors.joinError | 0xc001800348\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listzr65l9\" in resourcegroup=\"ea-list-z9h2z8\" to finish deleting: context canceled\n    ...\noccurred  \nfail [:0]: A node timeout occurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7f0e44476112ce4e675f687843407c03482d80d265239de5b2ddbd45c441a04e","failure_pattern":"failed to cleanup resource group: at least one HCP cluster \u003ccluster\u003e to delete: failed waiting for hcpcluster=\"\u003ccluster\u003e\" in resourcegroup=\"\u003cresource-group\u003e\" to finish deleting: context canceled","search_query":"failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"delayed-rbac-cluster\" in resourcegroup=\"delayed-rbac-kjlqnf\" to finish deleting: context canceled"}},{"row_id":"9be14794e7bb20bd5cde8693e18699917cc41845c9604856c97dda2db07ebc0d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly/2069193738747383808","occurred_at":"2026-06-22T23:00:05Z","failed_at":"e2e","signature_id":"efbd50b9b067f5293472532fae2e3ee43590f38d2028e4a53e161b1bfa71b808","test_name":"Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool \"np-1\" for private keyvault cluster \"private-kv-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0016aa030\u003e: \n    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-6r4vst, error: failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-6r4vst\" to finish creating: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool \"np-1\" for private keyvault cluster \"private-kv-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0016aa030\u003e: \n    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-6r4vst, error: failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-6r4vst\" to finish creating: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"941cbd5c6e3f3d8ed572893a689b2dfded75f418444a1a0d01baa9fc9e30d9c0","failure_pattern":"CreateNodePoolFromParam timed out after \u003cminutes\u003e minutes; context deadline exceeded","search_query":"timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-6r4vst, error: failed waiting for nodepool=\"np-1\" for cluster \"private-kv-cluster\" in resourcegroup=\"private-keyvault-6r4vst\" to finish creating: context deadline exceeded"}}],"failure_pattern_summary":{"signature_count":4,"failure_pattern_ids":["6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298","7f0e44476112ce4e675f687843407c03482d80d265239de5b2ddbd45c441a04e","941cbd5c6e3f3d8ed572893a689b2dfded75f418444a1a0d01baa9fc9e30d9c0"],"matched_occurrences":4,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"stg","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069105347804532736","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":false,"occurred_at":"2026-06-22T17:08:51Z"},"failed_test_count":0,"failure_pattern_summary":{"signature_count":0,"matched_occurrences":0,"unmatched_occurrences":0,"match_summary":"no_failures"}},{"run":{"environment":"stg","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069071579051986944","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-06-22T14:54:35Z"},"failed_at":["e2e"],"failed_test_count":2,"occurrences":[{"row_id":"b1a9b763c0b6fe3b15ae67ccc83ddff7495a4b023e5f42fcda45c40559c4d112","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069071579051986944","occurred_at":"2026-06-22T14:54:35Z","failed_at":"e2e","signature_id":"53021f426ac4e51d3cb6f45bee956b84c3f109e0d013a1d7786d79029ca52a86","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-8n2xlf/cluster-candidate-4-20-rb4p4z should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0012ae630\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster=\"cluster-candidate-4-20-rb4p4z\" in resourcegroup=\"rg-candidate-4-20-8n2xlf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-8n2xlf/cluster-candidate-4-20-rb4p4z should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0012ae630\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster=\"cluster-candidate-4-20-rb4p4z\" in resourcegroup=\"rg-candidate-4-20-8n2xlf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster=\"cluster-candidate-4-20-rb4p4z\" in resourcegroup=\"rg-candidate-4-20-8n2xlf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded"}},{"row_id":"ddcc33e7a2e97cf01fff27db3d0ab25e2778a3bf5dcfaa9a07771a7462e10eee","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069071579051986944","occurred_at":"2026-06-22T14:54:35Z","failed_at":"e2e","signature_id":"c8abacb525c80949e625916c338d8af12373d1f272cda34977ee1cabc43ca8e6","test_name":"Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:\nUnexpected error:\n    \u003c*errors.joinError | 0xc0010821f8\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4s6fx4\" to finish deleting: context canceled\n    ...\noccurred  \nfail [:0]: A node timeout occurred\nfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:\nUnexpected error:\n    \u003c*errors.joinError | 0xc0010821f8\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4s6fx4\" to finish deleting: context canceled\n    ...\noccurred  \nfail [:0]: A node timeout occurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7f0e44476112ce4e675f687843407c03482d80d265239de5b2ddbd45c441a04e","failure_pattern":"failed to cleanup resource group: at least one HCP cluster \u003ccluster\u003e to delete: failed waiting for hcpcluster=\"\u003ccluster\u003e\" in resourcegroup=\"\u003cresource-group\u003e\" to finish deleting: context canceled","search_query":"failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"delayed-rbac-cluster\" in resourcegroup=\"delayed-rbac-kjlqnf\" to finish deleting: context canceled"}}],"failure_pattern_summary":{"signature_count":2,"failure_pattern_ids":["6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298","7f0e44476112ce4e675f687843407c03482d80d265239de5b2ddbd45c441a04e"],"matched_occurrences":2,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"stg","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069002705430384640","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-06-22T10:20:59Z"},"failed_at":["e2e"],"failed_test_count":3,"occurrences":[{"row_id":"58dd5dee088093d2d65dd8c81ebd044a6ff132af4adaa2f1dad7ab71e1d957d8","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069002705430384640","occurred_at":"2026-06-22T10:20:59Z","failed_at":"e2e","signature_id":"cb0cbe311134b6ebcefbfb792a20a8cefb1dc9ab6e7dbee066b9898fc5752e2d","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-gn2h29/cluster-candidate-4-21-jmhjfc should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc001400a80\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-jmhjfc, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: failed waiting for cluster=\"cluster-candidate-4-21-jmhjfc\" in resourcegroup=\"rg-candidate-4-21-gn2h29\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-gn2h29/cluster-candidate-4-21-jmhjfc should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc001400a80\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-jmhjfc, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: failed waiting for cluster=\"cluster-candidate-4-21-jmhjfc\" in resourcegroup=\"rg-candidate-4-21-gn2h29\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster=\"cluster-candidate-4-20-rb4p4z\" in resourcegroup=\"rg-candidate-4-20-8n2xlf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded"}},{"row_id":"b0428e0a55923ec445f0f891f1352d67317f983e04eb563ce56bde28e8ceba5e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069002705430384640","occurred_at":"2026-06-22T10:20:59Z","failed_at":"e2e","signature_id":"7ef0493f254c06f948fbb27c1e5f409fe6455019f3ee1b4e053c6965c5af13ee","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster \"cluster-zstream-4-22-nhtnkl\" with version 4.22.0 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0015a5aa0\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-nhtnkl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: failed waiting for cluster=\"cluster-zstream-4-22-nhtnkl\" in resourcegroup=\"rg-zstream-upgrade-4-22-jdkmgf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster \"cluster-zstream-4-22-nhtnkl\" with version 4.22.0 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0015a5aa0\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-nhtnkl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: failed waiting for cluster=\"cluster-zstream-4-22-nhtnkl\" in resourcegroup=\"rg-zstream-upgrade-4-22-jdkmgf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster=\"cluster-candidate-4-20-rb4p4z\" in resourcegroup=\"rg-candidate-4-20-8n2xlf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded"}},{"row_id":"ccd2f5814c20ff169263b2bb76cba3009006fb2fcc4335aa9ef9645e42b614b1","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2069002705430384640","occurred_at":"2026-06-22T10:20:59Z","failed_at":"e2e","signature_id":"ea7f7ea2b3e749de6059661f9c0bd4c2ee23fa07bab3900cf551780158d30fd8","test_name":"Authorized CIDRs Connectivity should allow API access only from authorized VM IP","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:269]: failed to add password to app registration\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc000d45830\u003e: \n    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:269]: failed to add password to app registration\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc000d45830\u003e: \n    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"078c8370f997bf97070c98ca66b6e31b17e4874c4cf5b608a229a824de3b303d","failure_pattern":"add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API","search_query":"add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API"}}],"failure_pattern_summary":{"signature_count":3,"failure_pattern_ids":["078c8370f997bf97070c98ca66b6e31b17e4874c4cf5b608a229a824de3b303d","6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298"],"matched_occurrences":3,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"stg","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2068971202583465984","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-06-22T08:15:48Z"},"failed_at":["e2e"],"failed_test_count":1,"occurrences":[{"row_id":"21bea96c894d5105db010806e082205070b012b865730004e89b4ce48ea0138a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel/2068971202583465984","occurred_at":"2026-06-22T08:15:48Z","failed_at":"e2e","signature_id":"4e453a2e9360183edfba139347d397418ae5d38eff0fca154ae702111c3f49fa","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster \"cluster-zstream-4-20-xb84n7\" with version 4.20.25 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc000316570\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-xb84n7, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: failed waiting for cluster=\"cluster-zstream-4-20-xb84n7\" in resourcegroup=\"rg-zstream-upgrade-4-20-fn7428\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster \"cluster-zstream-4-20-xb84n7\" with version 4.20.25 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc000316570\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-xb84n7, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: failed waiting for cluster=\"cluster-zstream-4-20-xb84n7\" in resourcegroup=\"rg-zstream-upgrade-4-20-fn7428\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster=\"cluster-candidate-4-20-rb4p4z\" in resourcegroup=\"rg-candidate-4-20-8n2xlf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded"}}],"failure_pattern_summary":{"signature_count":1,"failure_pattern_ids":["6ed54686dd2f6d32054cbf144906e399a60d9c8b1a6d87d91850a99759063298"],"matched_occurrences":1,"unmatched_occurrences":0,"match_summary":"single_clustered"}},{"run":{"environment":"stg","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel/2068876744932200448","job_name":"periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-06-22T02:00:28Z"},"failed_at":["e2e"],"failed_test_count":1,"occurrences":[{"row_id":"76a4dd4b956da798a342159049d09e90a1c8c13f1603569c499ab3186e5d6115","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel/2068876744932200448","occurred_at":"2026-06-22T02:00:28Z","failed_at":"e2e","signature_id":"dc74b26189a451aa3afa2fb79f8978fccc65d244edcdd6b89d697215b555f044","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest","test_suite":"stage/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231]: failed to verify 3 ready and schedulable nodes for nodepool npupgrade-4-20 after upgrade\nExpected success, but got an error:\n    \u003c*errors.errorString | 0xc000f85a50\u003e: \n    expected 3 ready (and schedulable) nodes in node pool \"npupgrade-4-20\", found 4\n    ...\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231]: failed to verify 3 ready and schedulable nodes for nodepool npupgrade-4-20 after upgrade\nExpected success, but got an error:\n    \u003c*errors.errorString | 0xc000f85a50\u003e: \n    expected 3 ready (and schedulable) nodes in node pool \"npupgrade-4-20\", found 4\n    ...","failure_pattern_match":{"status":"clustered","failure_pattern_id":"b67cae5d7b1734c61859c2df1079b54448a4d5968fa6f6cea032c780834dc503","failure_pattern":"expected \u003ccount\u003e ready (and schedulable) nodes, found \u003ccount\u003e","search_query":"expected 3 ready (and schedulable) nodes in node pool \"npupgrade-4-20\", found 4"}}],"failure_pattern_summary":{"signature_count":1,"failure_pattern_ids":["b67cae5d7b1734c61859c2df1079b54448a4d5968fa6f6cea032c780834dc503"],"matched_occurrences":1,"unmatched_occurrences":0,"match_summary":"single_clustered"}}]}]}