{"meta":{"date":"2026-08-16","timezone":"UTC","generated_at":"2026-08-25T23:13:20Z","environments":["prod"]},"environments":[{"environment":"prod","summary":{"total_runs":2,"failed_runs":2,"runs_with_occurrences":2,"runs_with_failure_pattern_matches":2,"runs_with_unmatched_occurrences":0,"runs_without_occurrence_rows":0,"failed_runs_without_occurrence_rows":0},"runs":[{"run":{"environment":"prod","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly/2089125084324171776","job_name":"periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-16T23:00:08Z"},"failed_at":["e2e"],"failed_test_count":4,"occurrences":[{"row_id":"432f2aa5d9e6e7d9aa367a165595dc590f7f1f85ff60de890d762bf59f3e2bef","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly/2089125084324171776","occurred_at":"2026-08-16T23:00:08Z","failed_at":"e2e","signature_id":"7c67803c2e254d1bdc48291849f11c9089e77cade38fe4f574961e24bf00b185","test_name":"Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public","test_suite":"prod/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc001a62600\u003e: \n    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-228gln6xcsnr, error: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-228gln6xcsnr\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-228gln6xcsnr, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc001a62600\u003e: \n    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-228gln6xcsnr, error: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-228gln6xcsnr\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-228gln6xcsnr, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"120e48adced5265e3c99ccc28967cb2a7471c544770275711c2aee3f2d3418be","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed to create HCP cluster cluster-nightly-5-1-f2cpbk, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: failed waiting for cluster=\"cluster-nightly-5-1-f2cpbk\" in resourcegroup=\"rg-nightly-5-1-8lqmmddrt6vr\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: context deadline exceeded"}},{"row_id":"4b7bd386dfd235ab2160947e6137976f9d8a948a6b1aa8505f2f040270b97d5d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly/2089125084324171776","occurred_at":"2026-08-16T23:00:08Z","failed_at":"e2e","signature_id":"de4facbbe95a508fa08a98dfa3686efbd5c1d98a4b085f7f2d6521cbfcf56a90","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1","test_suite":"prod/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-5-1-8lqmmddrt6vr/cluster-nightly-5-1-f2cpbk should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0018ec570\u003e: \n    failed to create HCP cluster cluster-nightly-5-1-f2cpbk, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: failed waiting for cluster=\"cluster-nightly-5-1-f2cpbk\" in resourcegroup=\"rg-nightly-5-1-8lqmmddrt6vr\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-nightly-5-1-8lqmmddrt6vr/cluster-nightly-5-1-f2cpbk should provision\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0018ec570\u003e: \n    failed to create HCP cluster cluster-nightly-5-1-f2cpbk, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: failed waiting for cluster=\"cluster-nightly-5-1-f2cpbk\" in resourcegroup=\"rg-nightly-5-1-8lqmmddrt6vr\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"120e48adced5265e3c99ccc28967cb2a7471c544770275711c2aee3f2d3418be","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed to create HCP cluster cluster-nightly-5-1-f2cpbk, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: failed waiting for cluster=\"cluster-nightly-5-1-f2cpbk\" in resourcegroup=\"rg-nightly-5-1-8lqmmddrt6vr\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-f2cpbk in resource group rg-nightly-5-1-8lqmmddrt6vr, error: context deadline exceeded"}},{"row_id":"79f39eb8b9f2867071efc98dd656d360ebdca8d442bcebd5ecc5e1f16c31d70c","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly/2089125084324171776","occurred_at":"2026-08-16T23:00:08Z","failed_at":"e2e","signature_id":"0a2d5ab682d287e3a7c317d87bd1c4e5dc5a30cdcaf40caebdb11085f6f42e8d","test_name":"Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES","test_suite":"prod/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:163]: cluster verification failed\nUnexpected error:\n    \u003c*errors.joinError | 0xc000dc6330\u003e: \n    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-lcx67-w9nqc_OSDisk for VM des-encrypt-des-np-lcx67-w9nqc has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    disk des-encrypt-des-np-lcx67-wnckj_OSDisk for VM des-encrypt-des-np-lcx67-wnckj has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:163]: cluster verification failed\nUnexpected error:\n    \u003c*errors.joinError | 0xc000dc6330\u003e: \n    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-lcx67-w9nqc_OSDisk for VM des-encrypt-des-np-lcx67-w9nqc has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    disk des-encrypt-des-np-lcx67-wnckj_OSDisk for VM des-encrypt-des-np-lcx67-wnckj has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"4651710172f1dc3dcc71cbef1cab1532005f313f9d235f5a1fba54c8b7491ced","failure_pattern":"VerifyVMOSDiskCustomerEncryption(nodePool=\u003cnodepool\u003e) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-lcx67-w9nqc_OSDisk for VM des-encrypt-des-np-lcx67-w9nqc has encryption type EncryptionAtRestWithPlatformKey, expected","search_query":"VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-lcx67-w9nqc_OSDisk for VM des-encrypt-des-np-lcx67-w9nqc has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey"}},{"row_id":"7ae51a5f2f24b95d6ee861bc3c7704cd554cf521f85019fdc1b1a8ad609a9be9","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly/2089125084324171776","occurred_at":"2026-08-16T23:00:08Z","failed_at":"e2e","signature_id":"1fae781aa1436aa513c33238e437990551408d6e08ed6883fd2fc167df03858d","test_name":"Customer should be able to create a cluster with an external auth config and get the external auth config","test_suite":"prod/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:247]: failed to create console OAuth client secret external-auth-console-openshift-console\nUnexpected error:\n    \u003c*url.Error | 0xc002ba23f0\u003e: \n    Post \"https://api.ea-cluster.d8n6.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-config/secrets\": dial tcp: lookup api.ea-cluster.d8n6.uksouth.aroapp-hcp.io on 172.30.0.10:53: read udp 10.131.181.244:52227-\u003e172.30.0.10:53: i/o timeout\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:247]: failed to create console OAuth client secret external-auth-console-openshift-console\nUnexpected error:\n    \u003c*url.Error | 0xc002ba23f0\u003e: \n    Post \"https://api.ea-cluster.d8n6.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-config/secrets\": dial tcp: lookup api.ea-cluster.d8n6.uksouth.aroapp-hcp.io on 172.30.0.10:53: read udp 10.131.181.244:52227-\u003e172.30.0.10:53: i/o timeout\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"9c73f1a9954a7c79c8feb55d74392afe754a52da175b2b198ee22038ffd75ab2","failure_pattern":"Post \"\u003curl\u003e\": dial tcp: lookup \u003chost\u003e on \u003cdns-server\u003e: read udp \u003cip\u003e-\u003e\u003cip\u003e: i/o timeout","search_query":"Post \"https://api.ea-cluster.d8n6.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-config/secrets\": dial tcp: lookup api.ea-cluster.d8n6.uksouth.aroapp-hcp.io on 172.30.0.10:53: read udp 10.131.181.244:52227-\u003e172.30.0.10:53: i/o timeout"}}],"failure_pattern_summary":{"signature_count":4,"failure_pattern_ids":["120e48adced5265e3c99ccc28967cb2a7471c544770275711c2aee3f2d3418be","4651710172f1dc3dcc71cbef1cab1532005f313f9d235f5a1fba54c8b7491ced","9c73f1a9954a7c79c8feb55d74392afe754a52da175b2b198ee22038ffd75ab2"],"matched_occurrences":4,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"prod","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel/2088808202920529920","job_name":"periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-16T02:00:58Z"},"failed_at":["e2e"],"failed_test_count":1,"occurrences":[{"row_id":"cab5d46546b4de63aa1f46f70fa022da4e99739cd94f6ae73b776493dadabfc6","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel/2088808202920529920","occurred_at":"2026-08-16T02:00:58Z","failed_at":"e2e","signature_id":"fb86452a32eab4f29a42275f83daefeb5166e6c1798d574fea3352f02a6e1b50","test_name":"Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES","test_suite":"prod/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:163]: cluster verification failed\nUnexpected error:\n    \u003c*errors.joinError | 0xc00127ef48\u003e: \n    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-gtcnh-ltxb8_OSDisk for VM des-encrypt-des-np-gtcnh-ltxb8 has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    disk des-encrypt-des-np-gtcnh-wgdmc_OSDisk for VM des-encrypt-des-np-gtcnh-wgdmc has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:163]: cluster verification failed\nUnexpected error:\n    \u003c*errors.joinError | 0xc00127ef48\u003e: \n    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-gtcnh-ltxb8_OSDisk for VM des-encrypt-des-np-gtcnh-ltxb8 has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    disk des-encrypt-des-np-gtcnh-wgdmc_OSDisk for VM des-encrypt-des-np-gtcnh-wgdmc has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7481dd47ddce6042da41d4027eecd291d6730e2a016caa231132e83f1d6dbfd9","failure_pattern":"VerifyVMOSDiskCustomerEncryption(nodePool=\u003cnodepool\u003e) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-gtcnh-ltxb8_OSDisk for VM des-encrypt-des-np-gtcnh-ltxb8 has encryption type EncryptionAtRestWithPlatformKey, expected","search_query":"VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-gtcnh-ltxb8_OSDisk for VM des-encrypt-des-np-gtcnh-ltxb8 has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey"}}],"failure_pattern_summary":{"signature_count":1,"failure_pattern_ids":["7481dd47ddce6042da41d4027eecd291d6730e2a016caa231132e83f1d6dbfd9"],"matched_occurrences":1,"unmatched_occurrences":0,"match_summary":"single_clustered"}}]}]}