{"meta":{"date":"2026-08-07","timezone":"UTC","generated_at":"2026-08-26T22:17:27Z","environments":["int"]},"environments":[{"environment":"int","summary":{"total_runs":8,"failed_runs":8,"runs_with_occurrences":8,"runs_with_failure_pattern_matches":8,"runs_with_unmatched_occurrences":0,"runs_without_occurrence_rows":0,"failed_runs_without_occurrence_rows":0},"runs":[{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T23:58:58Z"},"failed_at":["e2e"],"failed_test_count":7,"occurrences":[{"row_id":"35f640dcecf4b7711c465f9858d6e3a8dc41efda121da20efd6cc33831e4e468","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","occurred_at":"2026-08-07T23:58:58Z","failed_at":"e2e","signature_id":"9f6f42bed68e6d3b86d81931d02f06532f74fc43be808bc0172db5fcca80bec4","test_name":"ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.002s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.002s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","failure_pattern":"cluster entered terminal Failed state after role assignment deployment","search_query":"cluster entered terminal Failed state after role assignment deployment"}},{"row_id":"414c866b627f80a9e1a0ff6048293cfc62b30168bdbab43ff35527c3c97a2139","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","occurred_at":"2026-08-07T23:58:58Z","failed_at":"e2e","signature_id":"3ba597a0a0cd92631c41dad889e1bcaf99d6196359258cc240532ea93cc7026d","test_name":"Customer should be able to create a HCP cluster and use cilium CNI plugin","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc00195a3a8\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-zltn98z6tdj9\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69abc17b-84d4-40b5-bcfb-891dad5f5ac1\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69abc17b-84d4-40b5-bcfb-891dad5f5ac1\",\n      \"name\": \"69abc17b-84d4-40b5-bcfb-891dad5f5ac1\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:55:37.832028004Z\",\n      \"endTime\": \"2026-08-08T01:19:43.636723642Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s22o8s1vf97eh5736e1ggfcssv0b54a still exists (deletion dispatched at 2026-08-08T00:55:39Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc00195a3a8\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-zltn98z6tdj9\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69abc17b-84d4-40b5-bcfb-891dad5f5ac1\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69abc17b-84d4-40b5-bcfb-891dad5f5ac1\",\n      \"name\": \"69abc17b-84d4-40b5-bcfb-891dad5f5ac1\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:55:37.832028004Z\",\n      \"endTime\": \"2026-08-08T01:19:43.636723642Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s22o8s1vf97eh5736e1ggfcssv0b54a still exists (deletion dispatched at 2026-08-08T00:55:39Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"86acf58965dfb7b96713f6a89ff381ba46ffdd44516e1b61638ec23fb893f80b","failure_pattern":"ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is \"uninstalling\"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"6ce2e094fab9cf58b277b7abb0ca76fac4888a41daa4ab44a41e26773767d441","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","occurred_at":"2026-08-07T23:58:58Z","failed_at":"e2e","signature_id":"8e069788cb14b52497959be5cd1c1e91635686f1aaf9ac547d7cc42368f1a858","test_name":"Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e8cc20\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-99xgf9mz5tb9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50fc85e3-3870-4309-8cff-cac58d6c2bef\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50fc85e3-3870-4309-8cff-cac58d6c2bef\",\n      \"name\": \"50fc85e3-3870-4309-8cff-cac58d6c2bef\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:38:23.395273244Z\",\n      \"endTime\": \"2026-08-08T00:57:32.681511281Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e8cc20\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-99xgf9mz5tb9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50fc85e3-3870-4309-8cff-cac58d6c2bef\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50fc85e3-3870-4309-8cff-cac58d6c2bef\",\n      \"name\": \"50fc85e3-3870-4309-8cff-cac58d6c2bef\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:38:23.395273244Z\",\n      \"endTime\": \"2026-08-08T00:57:32.681511281Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","failure_pattern":"ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"7179273b41d8b9b855782fbfef732dfc4c7fbbdcfc343a2b59dc82d0388f501f","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","occurred_at":"2026-08-07T23:58:58Z","failed_at":"e2e","signature_id":"c38b2bb8d5596b3a1e8995be6327c877b64e580f146636c34fd73c31630e3349","test_name":"Customer should be able to create an HCP cluster with back-level version 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc000210f30\u003e: \n    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-svf2cqd7v8s4, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-svf2cqd7v8s4, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc000210f30\u003e: \n    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-svf2cqd7v8s4, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-svf2cqd7v8s4, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"f90cae3228fe72b2427515e7c51834eebdc3e73c0b1aa6bca2abedd53a2e7578","failure_pattern":"DeleteHCPCluster timed out after \u003cminutes\u003e minutes; context deadline exceeded","search_query":"failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-svf2cqd7v8s4, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-svf2cqd7v8s4\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-svf2cqd7v8s4, error: context deadline exceeded"}},{"row_id":"7f7d7207e9cd17dfc42373fc5739c69fc09adf725b64cc7128fda1bc8f070d71","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","occurred_at":"2026-08-07T23:58:58Z","failed_at":"e2e","signature_id":"de48fd6ebd82051007a4d80a93f113aaafbb717f065665b23151c2b5c8a7ce65","test_name":"Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc00187a2e8\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-4rqwx6\" in resourcegroup=\"admin-credential-lifecycle-test-xtlwzpx9bhw8\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88105c69-59ef-4ce3-bda7-bf77c3dc139c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88105c69-59ef-4ce3-bda7-bf77c3dc139c\",\n      \"name\": \"88105c69-59ef-4ce3-bda7-bf77c3dc139c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:52:10.303231851Z\",\n      \"endTime\": \"2026-08-08T01:16:13.097660372Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s22o8ij989bsuf6l7v12p5uithckt93 still exists (deletion dispatched at 2026-08-08T00:52:10Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc00187a2e8\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-4rqwx6\" in resourcegroup=\"admin-credential-lifecycle-test-xtlwzpx9bhw8\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88105c69-59ef-4ce3-bda7-bf77c3dc139c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88105c69-59ef-4ce3-bda7-bf77c3dc139c\",\n      \"name\": \"88105c69-59ef-4ce3-bda7-bf77c3dc139c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:52:10.303231851Z\",\n      \"endTime\": \"2026-08-08T01:16:13.097660372Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s22o8ij989bsuf6l7v12p5uithckt93 still exists (deletion dispatched at 2026-08-08T00:52:10Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ef4dd01f86a43829d70263dd0a02bc0d1329f80083910a288015623f9842f040","failure_pattern":"ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is \"uninstalling\"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ab6ec46f8f501400e3d67e447d7d8d30bea608705299f38dcdd2d434a0a6e6d0","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","occurred_at":"2026-08-07T23:58:58Z","failed_at":"e2e","signature_id":"3b4fdd549183fb8ef99da52a3dcf177fb7a56e1ede7839b927df88cf8eb55645","test_name":"Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:93]: failed to create HCP cluster \"private-ingress\" with private ingress\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005bf160\u003e: \n    failed to create HCP cluster private-ingress: failed waiting for cluster=\"private-ingress\" in resourcegroup=\"private-ingress-sjfzqxprgc7n\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/346b118a-c1a4-48ca-b6a9-30a64c63256e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/346b118a-c1a4-48ca-b6a9-30a64c63256e\",\n      \"name\": \"346b118a-c1a4-48ca-b6a9-30a64c63256e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:36:39.309367694Z\",\n      \"endTime\": \"2026-08-08T00:55:42.448604058Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:93]: failed to create HCP cluster \"private-ingress\" with private ingress\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005bf160\u003e: \n    failed to create HCP cluster private-ingress: failed waiting for cluster=\"private-ingress\" in resourcegroup=\"private-ingress-sjfzqxprgc7n\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/346b118a-c1a4-48ca-b6a9-30a64c63256e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/346b118a-c1a4-48ca-b6a9-30a64c63256e\",\n      \"name\": \"346b118a-c1a4-48ca-b6a9-30a64c63256e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T00:36:39.309367694Z\",\n      \"endTime\": \"2026-08-08T00:55:42.448604058Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","failure_pattern":"ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"f08c98a60ecbf875244aa4148c40b10948317a4fd82478abbc8669f7f42f2270","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085878400177672192","occurred_at":"2026-08-07T23:58:58Z","failed_at":"e2e","signature_id":"f59cb127c740bc64bde390c75bfda24f3d892e518fac677b96a34a0ed8cafdaa","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc001184228\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-s6zj9t\" in resourcegroup=\"rg-zstream-upgrade-4-21-5b7hxtks6q64\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91ca2dc4-116d-47b5-bbf4-b471f46cb335\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91ca2dc4-116d-47b5-bbf4-b471f46cb335\",\n      \"name\": \"91ca2dc4-116d-47b5-bbf4-b471f46cb335\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T01:17:34.510452699Z\",\n      \"endTime\": \"2026-08-08T01:41:43.598195453Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s2378t1bpv4e98vqoorn8laj6rn7ahl still exists (deletion dispatched at 2026-08-08T01:17:35Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:\n    \u003c*errors.joinError | 0xc001184228\u003e: \n    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-s6zj9t\" in resourcegroup=\"rg-zstream-upgrade-4-21-5b7hxtks6q64\" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91ca2dc4-116d-47b5-bbf4-b471f46cb335\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/91ca2dc4-116d-47b5-bbf4-b471f46cb335\",\n      \"name\": \"91ca2dc4-116d-47b5-bbf4-b471f46cb335\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-08T01:17:34.510452699Z\",\n      \"endTime\": \"2026-08-08T01:41:43.598195453Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s2378t1bpv4e98vqoorn8laj6rn7ahl still exists (deletion dispatched at 2026-08-08T01:17:35Z); [clusterServiceStatus] ClusterService state is \\\"uninstalling\\\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ef4dd01f86a43829d70263dd0a02bc0d1329f80083910a288015623f9842f040","failure_pattern":"ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is \"uninstalling\"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}}],"failure_pattern_summary":{"signature_count":7,"failure_pattern_ids":["13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","86acf58965dfb7b96713f6a89ff381ba46ffdd44516e1b61638ec23fb893f80b","ef4dd01f86a43829d70263dd0a02bc0d1329f80083910a288015623f9842f040","f90cae3228fe72b2427515e7c51834eebdc3e73c0b1aa6bca2abedd53a2e7578"],"matched_occurrences":7,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085821660438466560","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T20:13:31Z"},"failed_at":["e2e"],"failed_test_count":5,"occurrences":[{"row_id":"0e6188fc4653810d2e5c09677e977e88e7d9d6a7c8302c22c6afe8115aa46f52","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085821660438466560","occurred_at":"2026-08-07T20:13:31Z","failed_at":"e2e","signature_id":"fabde1e6c314278347fbe022e55f7d769d9282d66487b818073a9f6c568d865f","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-cv9tkp2bfrlj/cluster-candidate-4-21-8hv5gr should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00108e8a0\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-8hv5gr: failed waiting for cluster=\"cluster-candidate-4-21-8hv5gr\" in resourcegroup=\"rg-candidate-4-21-cv9tkp2bfrlj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f076e731-be5f-40e0-89f6-e00eae2ce41b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f076e731-be5f-40e0-89f6-e00eae2ce41b\",\n      \"name\": \"f076e731-be5f-40e0-89f6-e00eae2ce41b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:54:21.72346339Z\",\n      \"endTime\": \"2026-08-07T21:13:24.985574515Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-cv9tkp2bfrlj/cluster-candidate-4-21-8hv5gr should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00108e8a0\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-8hv5gr: failed waiting for cluster=\"cluster-candidate-4-21-8hv5gr\" in resourcegroup=\"rg-candidate-4-21-cv9tkp2bfrlj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f076e731-be5f-40e0-89f6-e00eae2ce41b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f076e731-be5f-40e0-89f6-e00eae2ce41b\",\n      \"name\": \"f076e731-be5f-40e0-89f6-e00eae2ce41b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:54:21.72346339Z\",\n      \"endTime\": \"2026-08-07T21:13:24.985574515Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","failure_pattern":"ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"13efccd667f1871396583e985d4b0773b2de928d6aac4eb66ab68b645cf747e3","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085821660438466560","occurred_at":"2026-08-07T20:13:31Z","failed_at":"e2e","signature_id":"857c5f5ec59d4da00d9ed65b818cf4adf8198048ccf5ac0aa55de378b7234ffa","test_name":"FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00150f960\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-8p9f2zg2lfd9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dee5ed4b-65ee-4093-8e94-a89968b36a45\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00150f960\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-8p9f2zg2lfd9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dee5ed4b-65ee-4093-8e94-a89968b36a45\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f","failure_pattern":"ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: NoRegisteredProviderFound"}},{"row_id":"22cbe247e0bdda94e3590f848fd167c52bd39a3738780c13e47e048f3bbde218","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085821660438466560","occurred_at":"2026-08-07T20:13:31Z","failed_at":"e2e","signature_id":"99c10ed1810ea0f251e476d42a4847acb14c293c399ffd01675e6b30a4dd9349","test_name":"Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004bac40\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-j5mdml7j8d46\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa99c20d-bf89-404a-8ed3-73ce45aa3dd0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa99c20d-bf89-404a-8ed3-73ce45aa3dd0\",\n      \"name\": \"aa99c20d-bf89-404a-8ed3-73ce45aa3dd0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:33:34.297771936Z\",\n      \"endTime\": \"2026-08-07T20:52:34.143968254Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004bac40\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-j5mdml7j8d46\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa99c20d-bf89-404a-8ed3-73ce45aa3dd0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa99c20d-bf89-404a-8ed3-73ce45aa3dd0\",\n      \"name\": \"aa99c20d-bf89-404a-8ed3-73ce45aa3dd0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:33:34.297771936Z\",\n      \"endTime\": \"2026-08-07T20:52:34.143968254Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","failure_pattern":"ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"37cda3a689cf574462094a31322088fe4c585f8ca37042a45c8f9a0a91f69324","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085821660438466560","occurred_at":"2026-08-07T20:13:31Z","failed_at":"e2e","signature_id":"2afb94c446c9294ad15a77be0dce97a274fd3521c6754ae772d86b672254cce0","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-b84zljkl5jbq/cluster-candidate-4-22-5rc76r should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ab08e0\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-5rc76r: failed waiting for cluster=\"cluster-candidate-4-22-5rc76r\" in resourcegroup=\"rg-candidate-4-22-b84zljkl5jbq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba218375-f982-4fad-9c80-fb290534636b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba218375-f982-4fad-9c80-fb290534636b\",\n      \"name\": \"ba218375-f982-4fad-9c80-fb290534636b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:56:00.474630413Z\",\n      \"endTime\": \"2026-08-07T21:15:04.906578978Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-b84zljkl5jbq/cluster-candidate-4-22-5rc76r should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ab08e0\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-5rc76r: failed waiting for cluster=\"cluster-candidate-4-22-5rc76r\" in resourcegroup=\"rg-candidate-4-22-b84zljkl5jbq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba218375-f982-4fad-9c80-fb290534636b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba218375-f982-4fad-9c80-fb290534636b\",\n      \"name\": \"ba218375-f982-4fad-9c80-fb290534636b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:56:00.474630413Z\",\n      \"endTime\": \"2026-08-07T21:15:04.906578978Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","failure_pattern":"ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"e36e10b803c092bc61e50e14853680c8295479308542b37e4357a5a34cd61299","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085821660438466560","occurred_at":"2026-08-07T20:13:31Z","failed_at":"e2e","signature_id":"8fe4bcee38237924917b97b8ef4397b90f5a56f25a930ebb81e805fe78f82204","test_name":"Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000eb0e20\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-q8dcnzkbzbkg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5139453a-fe6a-479a-b54d-7dc371bbcab1\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5139453a-fe6a-479a-b54d-7dc371bbcab1\",\n      \"name\": \"5139453a-fe6a-479a-b54d-7dc371bbcab1\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:33:34.350855434Z\",\n      \"endTime\": \"2026-08-07T20:52:34.153152499Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000eb0e20\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-q8dcnzkbzbkg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5139453a-fe6a-479a-b54d-7dc371bbcab1\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5139453a-fe6a-479a-b54d-7dc371bbcab1\",\n      \"name\": \"5139453a-fe6a-479a-b54d-7dc371bbcab1\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T20:33:34.350855434Z\",\n      \"endTime\": \"2026-08-07T20:52:34.153152499Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","failure_pattern":"ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}}],"failure_pattern_summary":{"signature_count":5,"failure_pattern_ids":["13ca13618a2c4b8b7429b3001f7d02e419916c0961928bfe37af66198711655d","ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f"],"matched_occurrences":5,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085775639528345600","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T17:10:38Z"},"failed_at":["e2e"],"failed_test_count":1,"occurrences":[{"row_id":"7a898ffbcfe63d6ae042ca3cd8090db7bf9d7e2ba01f21d028ec9060e474b7b9","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085775639528345600","occurred_at":"2026-08-07T17:10:38Z","failed_at":"e2e","signature_id":"22fd0911ba4af1d0f409171f227272342169f9e1377e9c56c61079a5302b4964","test_name":"FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00164b280\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-8nmccl4vplhb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42d45e39-953e-403d-a22f-9a1d2fd0583c\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00164b280\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-8nmccl4vplhb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42d45e39-953e-403d-a22f-9a1d2fd0583c\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f","failure_pattern":"ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: NoRegisteredProviderFound"}}],"failure_pattern_summary":{"signature_count":1,"failure_pattern_ids":["ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f"],"matched_occurrences":1,"unmatched_occurrences":0,"match_summary":"single_clustered"}},{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T14:20:01Z"},"failed_at":["e2e"],"failed_test_count":45,"occurrences":[{"row_id":"002f1adf3209b82de352c7615389dcc49cdd24aaaf41ef01a56fcf0b57fdaf00","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"cf1effb1cb31448f745f6f3c89d9aed7a4ea306357491fc1e926f77e1433ec75","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-jf79gp\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00081ede0\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-jf79gp: failed waiting for cluster=\"cluster-zstream-4-22-jf79gp\" in resourcegroup=\"rg-zstream-upgrade-4-22-ppvwjcs6lrrs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0cfef7b-ad47-4881-9c7b-d7efd0ec0c0c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0cfef7b-ad47-4881-9c7b-d7efd0ec0c0c\",\n      \"name\": \"b0cfef7b-ad47-4881-9c7b-d7efd0ec0c0c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:17:15.473634321Z\",\n      \"endTime\": \"2026-08-07T15:36:17.616979971Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-jf79gp\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00081ede0\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-jf79gp: failed waiting for cluster=\"cluster-zstream-4-22-jf79gp\" in resourcegroup=\"rg-zstream-upgrade-4-22-ppvwjcs6lrrs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0cfef7b-ad47-4881-9c7b-d7efd0ec0c0c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0cfef7b-ad47-4881-9c7b-d7efd0ec0c0c\",\n      \"name\": \"b0cfef7b-ad47-4881-9c7b-d7efd0ec0c0c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:17:15.473634321Z\",\n      \"endTime\": \"2026-08-07T15:36:17.616979971Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"01ba28b16d5a3da063abdb80b54efd8909e6489627d37e8abab2117ad14a810e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"57767c9085cda99185d8b2323e7dd060e71afcee9fc819dae0e1d68a10503177","test_name":"Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008807a0\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-ht974mqw7fnz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0327a4b3-b2ee-4efe-a6cd-e2dd1dcf7f24\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0327a4b3-b2ee-4efe-a6cd-e2dd1dcf7f24\",\n      \"name\": \"0327a4b3-b2ee-4efe-a6cd-e2dd1dcf7f24\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:55:48.396578628Z\",\n      \"endTime\": \"2026-08-07T15:14:56.174236248Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008807a0\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-ht974mqw7fnz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0327a4b3-b2ee-4efe-a6cd-e2dd1dcf7f24\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0327a4b3-b2ee-4efe-a6cd-e2dd1dcf7f24\",\n      \"name\": \"0327a4b3-b2ee-4efe-a6cd-e2dd1dcf7f24\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:55:48.396578628Z\",\n      \"endTime\": \"2026-08-07T15:14:56.174236248Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"04316d853bdfec2eb3abbe67f439b8c7d0a1fd85181c7c07e25682d4ca2a5178","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"187020d22a0a1249bf570e649c7eb6df21f1e3f1aeead62863d08078844c8a9f","test_name":"Customer should be able to create an HCP cluster with back-level version 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:151]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc00178a570\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-wh5bnktd6mq4\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-wh5bnktd6mq4, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:151]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc00178a570\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-wh5bnktd6mq4\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-wh5bnktd6mq4, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-br948jjgf9df\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-br948jjgf9df, error: context deadline exceeded"}},{"row_id":"0530733865bf6ed4d177419f8f919d54aa6308a133f5e7c515500851b03681cf","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"6b0e2201aeaa08c7f8057dd3a349f9074c104b3bdf1af0d88ffd0f31bf454202","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-8cnxxk with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000edce40\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-8cnxxk: failed waiting for cluster=\"np-version-upgrade-cluster-8cnxxk\" in resourcegroup=\"rg-np-version-upgrade-8cnxxk-wpfc6ql7vktt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49311ef6-a44a-4964-aabe-0e241e36d260\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49311ef6-a44a-4964-aabe-0e241e36d260\",\n      \"name\": \"49311ef6-a44a-4964-aabe-0e241e36d260\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:20:08.120968358Z\",\n      \"endTime\": \"2026-08-07T15:39:17.863663712Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-8cnxxk with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000edce40\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-8cnxxk: failed waiting for cluster=\"np-version-upgrade-cluster-8cnxxk\" in resourcegroup=\"rg-np-version-upgrade-8cnxxk-wpfc6ql7vktt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49311ef6-a44a-4964-aabe-0e241e36d260\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/49311ef6-a44a-4964-aabe-0e241e36d260\",\n      \"name\": \"49311ef6-a44a-4964-aabe-0e241e36d260\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:20:08.120968358Z\",\n      \"endTime\": \"2026-08-07T15:39:17.863663712Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"086ca837cd590c172ba5b741d3a86d7e4283283446fd150498b17b5c33a6f2fe","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"e3fd952cfa4c54cec5ae259d55347610ab6d5b24bb16481ce833d9f93d6284d0","test_name":"Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc00071b1d0\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/423c6526-611a-4ccf-a67f-41aba9ad6484\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/423c6526-611a-4ccf-a67f-41aba9ad6484\",\n      \"name\": \"423c6526-611a-4ccf-a67f-41aba9ad6484\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:55:06.282226561Z\",\n      \"endTime\": \"2026-08-07T15:14:16.28205182Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc00071b1d0\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/423c6526-611a-4ccf-a67f-41aba9ad6484\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/423c6526-611a-4ccf-a67f-41aba9ad6484\",\n      \"name\": \"423c6526-611a-4ccf-a67f-41aba9ad6484\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:55:06.282226561Z\",\n      \"endTime\": \"2026-08-07T15:14:16.28205182Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0d2c775a9cc37c6fd4848a8b2bb29472fd07bf2f10327a8f7d7f0ce298e4fce8","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"47d3cad25bc7d0b6aa2494c3a4bdf6cf08d14f4a89a9b02b2454e7a7c65b752b","test_name":"Customer should be able to create a cluster with an external auth config and get the external auth config","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008b2580\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-qd5ctfrf72hw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9b4678d4-294c-408c-8008-fc0a7e93a846\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9b4678d4-294c-408c-8008-fc0a7e93a846\",\n      \"name\": \"9b4678d4-294c-408c-8008-fc0a7e93a846\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:18:31.098909772Z\",\n      \"endTime\": \"2026-08-07T15:37:37.858517424Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008b2580\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-qd5ctfrf72hw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9b4678d4-294c-408c-8008-fc0a7e93a846\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9b4678d4-294c-408c-8008-fc0a7e93a846\",\n      \"name\": \"9b4678d4-294c-408c-8008-fc0a7e93a846\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:18:31.098909772Z\",\n      \"endTime\": \"2026-08-07T15:37:37.858517424Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0f375340401b48f3a3d96a17bd34f57d27fb266a6db164082d30fe7dfc6df6ce","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"8d58c2e56c47f1ce4e8ef770229ae4ce4a2a711f3b4669a8f9cdd3c172e71db6","test_name":"FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f29be0\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-2klqp9mm4p4l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1403563f-0e25-4b88-9175-24229e4a4558\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f29be0\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-2klqp9mm4p4l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1403563f-0e25-4b88-9175-24229e4a4558\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f","failure_pattern":"ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: NoRegisteredProviderFound"}},{"row_id":"1184c720b79296a455dbf7fd6bdaefab167b2545b3a693e74513e38b938c190a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"9d24665dcba60650b691658bb4f482eace81d64ee7d2ac489702eb25773b67a7","test_name":"Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009fc760\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-29j4zbvc8cgd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cd6c227-900c-4884-a85e-b1834fccb619\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cd6c227-900c-4884-a85e-b1834fccb619\",\n      \"name\": \"4cd6c227-900c-4884-a85e-b1834fccb619\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.226694182Z\",\n      \"endTime\": \"2026-08-07T15:13:26.041923325Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009fc760\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-29j4zbvc8cgd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cd6c227-900c-4884-a85e-b1834fccb619\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cd6c227-900c-4884-a85e-b1834fccb619\",\n      \"name\": \"4cd6c227-900c-4884-a85e-b1834fccb619\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.226694182Z\",\n      \"endTime\": \"2026-08-07T15:13:26.041923325Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"132f01bfe0ea4ebb06b2bdf228d64babb7d5147fef6985cb047134c8c58f96b5","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"d640017ca9fcd362c4f7ac7f0be89492904ac2882bd6a6e9263865753f0ae664","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-gxnjjdz4nbxx/cluster-candidate-4-21-8728m4 should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006391c0\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-8728m4: failed waiting for cluster=\"cluster-candidate-4-21-8728m4\" in resourcegroup=\"rg-candidate-4-21-gxnjjdz4nbxx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba64e822-db2e-4c26-9e88-a7ef72949b21\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba64e822-db2e-4c26-9e88-a7ef72949b21\",\n      \"name\": \"ba64e822-db2e-4c26-9e88-a7ef72949b21\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:31.101892728Z\",\n      \"endTime\": \"2026-08-07T15:35:37.641221472Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-gxnjjdz4nbxx/cluster-candidate-4-21-8728m4 should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006391c0\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-8728m4: failed waiting for cluster=\"cluster-candidate-4-21-8728m4\" in resourcegroup=\"rg-candidate-4-21-gxnjjdz4nbxx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba64e822-db2e-4c26-9e88-a7ef72949b21\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba64e822-db2e-4c26-9e88-a7ef72949b21\",\n      \"name\": \"ba64e822-db2e-4c26-9e88-a7ef72949b21\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:31.101892728Z\",\n      \"endTime\": \"2026-08-07T15:35:37.641221472Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"15a29d6f792ea5c0b69be617fe4cb6a6140811d8128a0354c584069e322a2e8d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"f61c9994c95155531c9c2fcc7573f3d10ee962be24c72e9d12a24e06d2a3d263","test_name":"Customer should be able to create a HCP cluster and use cilium CNI plugin","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0016a30a0\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-xrm2p66vkqqc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948c25c2-f5b5-416f-9b9d-9c604c310d51\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948c25c2-f5b5-416f-9b9d-9c604c310d51\",\n      \"name\": \"948c25c2-f5b5-416f-9b9d-9c604c310d51\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.919287249Z\",\n      \"endTime\": \"2026-08-07T15:13:16.205386943Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0016a30a0\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-xrm2p66vkqqc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948c25c2-f5b5-416f-9b9d-9c604c310d51\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948c25c2-f5b5-416f-9b9d-9c604c310d51\",\n      \"name\": \"948c25c2-f5b5-416f-9b9d-9c604c310d51\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.919287249Z\",\n      \"endTime\": \"2026-08-07T15:13:16.205386943Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"2bd7baf86d1ee7df91ad358105ecb4fa39679f6874b6819dfb0ee8c607b2b9ef","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"a47a5bde18bf6dc2929798e099ee467d4908fb016d34f340bb9ae7206fde9a64","test_name":"Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0001ab040\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-vn5vkp57zl7h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e867da56-a904-48c9-a2f4-6ec60259ff0e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e867da56-a904-48c9-a2f4-6ec60259ff0e\",\n      \"name\": \"e867da56-a904-48c9-a2f4-6ec60259ff0e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:49.237163687Z\",\n      \"endTime\": \"2026-08-07T15:13:55.978734105Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0001ab040\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-vn5vkp57zl7h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e867da56-a904-48c9-a2f4-6ec60259ff0e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e867da56-a904-48c9-a2f4-6ec60259ff0e\",\n      \"name\": \"e867da56-a904-48c9-a2f4-6ec60259ff0e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:49.237163687Z\",\n      \"endTime\": \"2026-08-07T15:13:55.978734105Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"2f877501decc54bf8e2679f4a5e69623c9d300efffcf9d2da3333652ec6cde64","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"14c80964b9874c9081110be783c8e717ab9db60e5a66ec45327b015be13e84aa","test_name":"Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed\nfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed","failure_pattern_match":{"status":"clustered","failure_pattern_id":"e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b","failure_pattern":"Cluster provisioning failed","search_query":"Cluster provisioning failed"}},{"row_id":"3f83f41dfaba7cbe279620a8769c97422c942ee6bc4bfcde4d7a76a359158845","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"de12bc874b71b3b9769fdb9d192543a810c928d5d115817a5cc9f3f448001884","test_name":"Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000857520\u003e: \n    failed to create HCP cluster np-dg-minor-mbfl8p: failed waiting for cluster=\"np-dg-minor-mbfl8p\" in resourcegroup=\"rg-np-dg-minor-mbfl8p-wsw4qw2qpbks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d42a1888-d4d3-49a7-afe2-604910c390c8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d42a1888-d4d3-49a7-afe2-604910c390c8\",\n      \"name\": \"d42a1888-d4d3-49a7-afe2-604910c390c8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:35.919958221Z\",\n      \"endTime\": \"2026-08-07T15:58:36.84774215Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000857520\u003e: \n    failed to create HCP cluster np-dg-minor-mbfl8p: failed waiting for cluster=\"np-dg-minor-mbfl8p\" in resourcegroup=\"rg-np-dg-minor-mbfl8p-wsw4qw2qpbks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d42a1888-d4d3-49a7-afe2-604910c390c8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d42a1888-d4d3-49a7-afe2-604910c390c8\",\n      \"name\": \"d42a1888-d4d3-49a7-afe2-604910c390c8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:35.919958221Z\",\n      \"endTime\": \"2026-08-07T15:58:36.84774215Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"45e3529282de552c29f65187062a1b702a063dcfde9e2420054c669ab51de436","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"69bd66b6d04fbda55431cca5ce2192616e394de058d01d12e657d8cad6c4e68d","test_name":"ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.001s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.001s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","failure_pattern":"cluster entered terminal Failed state after role assignment deployment","search_query":"cluster entered terminal Failed state after role assignment deployment"}},{"row_id":"4ac131f870fabae6bd2fd63d9da5ebccdedb2ae538c5104cd577e119132874bf","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"027f446b1c0dc289deb12b8a638b955ca90504ca1c4faff5cbcdf2acb1e01693","test_name":"Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-cnjgmq with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ea0ca0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-cnjgmq: failed waiting for cluster=\"np-version-upgrade-cluster-cnjgmq\" in resourcegroup=\"rg-np-version-upgrade-cnjgmq-wtj7phhd7t7p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5736412-08cd-4700-b460-89b5905be844\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5736412-08cd-4700-b460-89b5905be844\",\n      \"name\": \"f5736412-08cd-4700-b460-89b5905be844\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:38:51.207478348Z\",\n      \"endTime\": \"2026-08-07T15:57:56.841927968Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-cnjgmq with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ea0ca0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-cnjgmq: failed waiting for cluster=\"np-version-upgrade-cluster-cnjgmq\" in resourcegroup=\"rg-np-version-upgrade-cnjgmq-wtj7phhd7t7p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5736412-08cd-4700-b460-89b5905be844\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5736412-08cd-4700-b460-89b5905be844\",\n      \"name\": \"f5736412-08cd-4700-b460-89b5905be844\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:38:51.207478348Z\",\n      \"endTime\": \"2026-08-07T15:57:56.841927968Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4c4e8f39f7ba81769b8d3babd7fce725b2e9986a1764a926415ea1f3795125e0","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"dc89e3062c93754745bbad490658ec77f06ea112c5c63248ddbffd95def1e83d","test_name":"Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-grhwwr at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f28e20\u003e: \n    failed to create HCP cluster np-skip-minor-grhwwr: failed waiting for cluster=\"np-skip-minor-grhwwr\" in resourcegroup=\"rg-np-skip-minor-grhwwr-frkq9mdn2t79\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/71461586-b128-4a10-b289-e839d2078eee\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/71461586-b128-4a10-b289-e839d2078eee\",\n      \"name\": \"71461586-b128-4a10-b289-e839d2078eee\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:15.355399625Z\",\n      \"endTime\": \"2026-08-07T15:58:16.838231606Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-grhwwr at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f28e20\u003e: \n    failed to create HCP cluster np-skip-minor-grhwwr: failed waiting for cluster=\"np-skip-minor-grhwwr\" in resourcegroup=\"rg-np-skip-minor-grhwwr-frkq9mdn2t79\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/71461586-b128-4a10-b289-e839d2078eee\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/71461586-b128-4a10-b289-e839d2078eee\",\n      \"name\": \"71461586-b128-4a10-b289-e839d2078eee\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:15.355399625Z\",\n      \"endTime\": \"2026-08-07T15:58:16.838231606Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4e7ab70506cacc2af5d2d0e24c79460d899bfda5c281c601ea44527f0aba96b1","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"908b9dc4a67ca0f816383d75ed1020573ff90b710ec4ad30f7d9126694e3885c","test_name":"Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004b04c0\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-z76r8vldx2wv\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/341b0b06-88a4-421c-bb07-c6191b228057\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/341b0b06-88a4-421c-bb07-c6191b228057\",\n      \"name\": \"341b0b06-88a4-421c-bb07-c6191b228057\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:58:01.88121463Z\",\n      \"endTime\": \"2026-08-07T15:17:04.693106775Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004b04c0\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-z76r8vldx2wv\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/341b0b06-88a4-421c-bb07-c6191b228057\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/341b0b06-88a4-421c-bb07-c6191b228057\",\n      \"name\": \"341b0b06-88a4-421c-bb07-c6191b228057\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:58:01.88121463Z\",\n      \"endTime\": \"2026-08-07T15:17:04.693106775Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"5424ecc7d03bca1ed97c7af273de8178ac084d56d364a87beb02965567df5492","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"d0ed6bd0531bb1f9bb7faa5231b9846d48fb6e807d4bc976304e7b918778e998","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-7bfg4cpclrtr/cluster-candidate-4-22-p4lqwv should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f7c460\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-p4lqwv: failed waiting for cluster=\"cluster-candidate-4-22-p4lqwv\" in resourcegroup=\"rg-candidate-4-22-7bfg4cpclrtr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/36bc8590-c5aa-4c6b-94c5-6cf939cc40f8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/36bc8590-c5aa-4c6b-94c5-6cf939cc40f8\",\n      \"name\": \"36bc8590-c5aa-4c6b-94c5-6cf939cc40f8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:22.957187308Z\",\n      \"endTime\": \"2026-08-07T15:35:28.185497199Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-7bfg4cpclrtr/cluster-candidate-4-22-p4lqwv should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f7c460\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-p4lqwv: failed waiting for cluster=\"cluster-candidate-4-22-p4lqwv\" in resourcegroup=\"rg-candidate-4-22-7bfg4cpclrtr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/36bc8590-c5aa-4c6b-94c5-6cf939cc40f8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/36bc8590-c5aa-4c6b-94c5-6cf939cc40f8\",\n      \"name\": \"36bc8590-c5aa-4c6b-94c5-6cf939cc40f8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:22.957187308Z\",\n      \"endTime\": \"2026-08-07T15:35:28.185497199Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"58e0b63b064d3ee1183cab38b91b53658b6d9798ebc1f854a123af9449186883","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"99628b4cd26e05c1fb3a0842ef27be0a645fb3a044f3d8ef34561b1b443d7f02","test_name":"Customer should create an HCP cluster and validate TLS certificates","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000715ac0\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-wpn25kjprg55\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e92d2a6-a7b4-4d67-9521-547a6c5cf743\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e92d2a6-a7b4-4d67-9521-547a6c5cf743\",\n      \"name\": \"8e92d2a6-a7b4-4d67-9521-547a6c5cf743\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:56:48.953933026Z\",\n      \"endTime\": \"2026-08-07T15:15:54.173332426Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000715ac0\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-wpn25kjprg55\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e92d2a6-a7b4-4d67-9521-547a6c5cf743\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e92d2a6-a7b4-4d67-9521-547a6c5cf743\",\n      \"name\": \"8e92d2a6-a7b4-4d67-9521-547a6c5cf743\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:56:48.953933026Z\",\n      \"endTime\": \"2026-08-07T15:15:54.173332426Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"59872b17dc877e0a141e5cae70b871abccdd25948e9fd3af2944b8dd081ae454","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"cd2cc2e9c5ccd580d9e76a224706e6c631ede4c1da319db8f761aed6fce7592b","test_name":"Customer should not be able to reuse subnets and NSGs between clusters","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008fb340\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-9fm9d6x8g5z2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4f81f17-054e-48ed-b358-537fe09f92bc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4f81f17-054e-48ed-b358-537fe09f92bc\",\n      \"name\": \"e4f81f17-054e-48ed-b358-537fe09f92bc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.274353729Z\",\n      \"endTime\": \"2026-08-07T15:13:25.968892058Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008fb340\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-9fm9d6x8g5z2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4f81f17-054e-48ed-b358-537fe09f92bc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4f81f17-054e-48ed-b358-537fe09f92bc\",\n      \"name\": \"e4f81f17-054e-48ed-b358-537fe09f92bc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.274353729Z\",\n      \"endTime\": \"2026-08-07T15:13:25.968892058Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"5d19378f69b19a3cf58a186625d02162d4fec6a8da32b06154c2b02d28f54978","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"2b21e290732895e01009644ce304946b17cf17031196d555ce4e97c501a825d7","test_name":"Customer should be able to create an HCP cluster and manage pull secrets","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00172b180\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kx5v5mm55vgm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5004e964-a313-4aad-a3c7-5853a9f5b08c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5004e964-a313-4aad-a3c7-5853a9f5b08c\",\n      \"name\": \"5004e964-a313-4aad-a3c7-5853a9f5b08c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.463176064Z\",\n      \"endTime\": \"2026-08-07T15:13:16.175125544Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00172b180\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kx5v5mm55vgm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5004e964-a313-4aad-a3c7-5853a9f5b08c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5004e964-a313-4aad-a3c7-5853a9f5b08c\",\n      \"name\": \"5004e964-a313-4aad-a3c7-5853a9f5b08c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:07.463176064Z\",\n      \"endTime\": \"2026-08-07T15:13:16.175125544Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"696c38617951ae7feba50ebb44ef9399b4c5e5f26d1b74adbb9482d27a12707b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"4b76e186310bcb07c0bcb192c5f2a1765b31eeca09563181187e9f783bd9aa35","test_name":"Customer should not be able to perform various invalid operations on cluster resources","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000696b40\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-v5j4rz7mjxrr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e8dcb081-0262-4d19-86be-638a020708ac\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e8dcb081-0262-4d19-86be-638a020708ac\",\n      \"name\": \"e8dcb081-0262-4d19-86be-638a020708ac\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:40:09.946987189Z\",\n      \"endTime\": \"2026-08-07T15:59:16.817572922Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000696b40\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-v5j4rz7mjxrr\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e8dcb081-0262-4d19-86be-638a020708ac\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e8dcb081-0262-4d19-86be-638a020708ac\",\n      \"name\": \"e8dcb081-0262-4d19-86be-638a020708ac\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:40:09.946987189Z\",\n      \"endTime\": \"2026-08-07T15:59:16.817572922Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"6f82ded9e567a99900ac72a4d233ecd31a7b495b1042146934698f800f1585c4","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"70de3a4487a4a2b809f9f2408e3aeda01da8d2b0ab2f053e408f16da9b300012","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-zjln9w\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e88c40\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-zjln9w: failed waiting for cluster=\"cluster-zstream-4-20-zjln9w\" in resourcegroup=\"rg-zstream-upgrade-4-20-mvvqmq577c4s\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ce97c91-e1d2-42dd-970d-09d1b43dc82d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ce97c91-e1d2-42dd-970d-09d1b43dc82d\",\n      \"name\": \"1ce97c91-e1d2-42dd-970d-09d1b43dc82d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:47.473612764Z\",\n      \"endTime\": \"2026-08-07T15:35:47.81308082Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-zjln9w\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e88c40\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-zjln9w: failed waiting for cluster=\"cluster-zstream-4-20-zjln9w\" in resourcegroup=\"rg-zstream-upgrade-4-20-mvvqmq577c4s\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ce97c91-e1d2-42dd-970d-09d1b43dc82d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ce97c91-e1d2-42dd-970d-09d1b43dc82d\",\n      \"name\": \"1ce97c91-e1d2-42dd-970d-09d1b43dc82d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:47.473612764Z\",\n      \"endTime\": \"2026-08-07T15:35:47.81308082Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"71a1d246d6eac753902bb7f105f36eeca0af4627b32c2c4c289b08a97913fa19","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"6da2dacbec82d202b9abdba714d1cd4a992aa330df4f277b53491193b547c877","test_name":"Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a10320\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rz9sc2h2p9sh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/079763bf-9a81-41c0-96a1-c7e36c9a7147\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/079763bf-9a81-41c0-96a1-c7e36c9a7147\",\n      \"name\": \"079763bf-9a81-41c0-96a1-c7e36c9a7147\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:57:44.942213597Z\",\n      \"endTime\": \"2026-08-07T15:16:44.419503562Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a10320\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rz9sc2h2p9sh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/079763bf-9a81-41c0-96a1-c7e36c9a7147\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/079763bf-9a81-41c0-96a1-c7e36c9a7147\",\n      \"name\": \"079763bf-9a81-41c0-96a1-c7e36c9a7147\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:57:44.942213597Z\",\n      \"endTime\": \"2026-08-07T15:16:44.419503562Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"790db729d8526957c3c79dc9765b1cfaf765a51c17837ea687f79cfa56823dbf","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"0bacb8e5caaaf0eb0e31dcb07f000868915134897270a579da16541f38fd7e70","test_name":"HCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a8f140\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-sjjtnfcghj5c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b33dcd9-fe8a-46ec-9058-9b49a02bf22a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b33dcd9-fe8a-46ec-9058-9b49a02bf22a\",\n      \"name\": \"8b33dcd9-fe8a-46ec-9058-9b49a02bf22a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:18:31.335460013Z\",\n      \"endTime\": \"2026-08-07T15:37:37.897869Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a8f140\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-sjjtnfcghj5c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b33dcd9-fe8a-46ec-9058-9b49a02bf22a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b33dcd9-fe8a-46ec-9058-9b49a02bf22a\",\n      \"name\": \"8b33dcd9-fe8a-46ec-9058-9b49a02bf22a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:18:31.335460013Z\",\n      \"endTime\": \"2026-08-07T15:37:37.897869Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"79cc4d5ba5edfd8e0c3300663e6adca65de73074e016c199d2c511a39acde72b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"9dd974181ce1745c6e663a38fb74acc35bc96901bb2bec62bfc5655b7fd51e18","test_name":"Customer should be able to use workload identity via the cluster OIDC issuer URL","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00085cc00\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q5zv9p5hqkr9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f71fd88-b5d3-4594-a6cc-142f5be4b807\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f71fd88-b5d3-4594-a6cc-142f5be4b807\",\n      \"name\": \"1f71fd88-b5d3-4594-a6cc-142f5be4b807\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:26.071222886Z\",\n      \"endTime\": \"2026-08-07T15:58:26.891970379Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00085cc00\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q5zv9p5hqkr9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f71fd88-b5d3-4594-a6cc-142f5be4b807\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1f71fd88-b5d3-4594-a6cc-142f5be4b807\",\n      \"name\": \"1f71fd88-b5d3-4594-a6cc-142f5be4b807\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:26.071222886Z\",\n      \"endTime\": \"2026-08-07T15:58:26.891970379Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"80093e429079506e824e7cddf1bf26342487d4b8107930b8915506bc5406a0fa","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"b565d360f54944005de6a5907e54a32ce37826494481e065d0b22cd8c9804fae","test_name":"Customer should respect cluster-wide node limits with nodepool autoscaling","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0015846e0\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qwbf9knlvsbb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb26cfd3-5403-40c2-9a78-1f75342279fb\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb26cfd3-5403-40c2-9a78-1f75342279fb\",\n      \"name\": \"eb26cfd3-5403-40c2-9a78-1f75342279fb\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:19.924672812Z\",\n      \"endTime\": \"2026-08-07T15:38:27.966997196Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0015846e0\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qwbf9knlvsbb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb26cfd3-5403-40c2-9a78-1f75342279fb\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb26cfd3-5403-40c2-9a78-1f75342279fb\",\n      \"name\": \"eb26cfd3-5403-40c2-9a78-1f75342279fb\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:19.924672812Z\",\n      \"endTime\": \"2026-08-07T15:38:27.966997196Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"88edb2f437d6b8fd650ffcd0355b682d84885c1eed5f93d5ad4ed4cb42309fec","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"2efdcb6cd8ba0372479850e66cb5dd3a1ee4b8b4357680c9b2417f62068e504e","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-zqss2x\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f3eca0\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-zqss2x: failed waiting for cluster=\"cluster-zstream-4-21-zqss2x\" in resourcegroup=\"rg-zstream-upgrade-4-21-nkxw9pwkqdsh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4526eaa-2ef5-4ddd-b93d-9cc2f5543663\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4526eaa-2ef5-4ddd-b93d-9cc2f5543663\",\n      \"name\": \"e4526eaa-2ef5-4ddd-b93d-9cc2f5543663\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:47.58964132Z\",\n      \"endTime\": \"2026-08-07T15:35:47.647931236Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-zqss2x\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f3eca0\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-zqss2x: failed waiting for cluster=\"cluster-zstream-4-21-zqss2x\" in resourcegroup=\"rg-zstream-upgrade-4-21-nkxw9pwkqdsh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4526eaa-2ef5-4ddd-b93d-9cc2f5543663\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e4526eaa-2ef5-4ddd-b93d-9cc2f5543663\",\n      \"name\": \"e4526eaa-2ef5-4ddd-b93d-9cc2f5543663\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:47.58964132Z\",\n      \"endTime\": \"2026-08-07T15:35:47.647931236Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"a2f40cbfff822379367a91a6860bded2cde42c44a2077fbe7f65a44af4113319","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"219c69fa52da2618fb214f674de589379eab22d022a801f0123635c0197122b8","test_name":"Customer should be able to lifecycle and confirm external auth on a cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e5f740\u003e: \n    failed to create HCP cluster ea-listzwl9ct: failed waiting for cluster=\"ea-listzwl9ct\" in resourcegroup=\"ea-list-zdpbrrskw4vx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1988d26f-44dd-4379-96e7-cb6af4b1c8f6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1988d26f-44dd-4379-96e7-cb6af4b1c8f6\",\n      \"name\": \"1988d26f-44dd-4379-96e7-cb6af4b1c8f6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:18:41.549158767Z\",\n      \"endTime\": \"2026-08-07T15:37:47.949405375Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e5f740\u003e: \n    failed to create HCP cluster ea-listzwl9ct: failed waiting for cluster=\"ea-listzwl9ct\" in resourcegroup=\"ea-list-zdpbrrskw4vx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1988d26f-44dd-4379-96e7-cb6af4b1c8f6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1988d26f-44dd-4379-96e7-cb6af4b1c8f6\",\n      \"name\": \"1988d26f-44dd-4379-96e7-cb6af4b1c8f6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:18:41.549158767Z\",\n      \"endTime\": \"2026-08-07T15:37:47.949405375Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"a5caa5e14138f23aea80499915a6d3f35b3552ced136355f4cc25c36ea96f689","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"31379494d3365ab9b19c5525a9ee5aeb8e08fa872b94e289f5a4dbcb5ffaf1f2","test_name":"Customer should be able to create an HCP cluster and manage ImageDigestMirrors","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006328c0\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jx5cqwz25kk7\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/44ab5ef7-11ac-498c-a686-889f791bd53d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/44ab5ef7-11ac-498c-a686-889f791bd53d\",\n      \"name\": \"44ab5ef7-11ac-498c-a686-889f791bd53d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:02.012372629Z\",\n      \"endTime\": \"2026-08-07T15:38:07.880331415Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006328c0\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jx5cqwz25kk7\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/44ab5ef7-11ac-498c-a686-889f791bd53d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/44ab5ef7-11ac-498c-a686-889f791bd53d\",\n      \"name\": \"44ab5ef7-11ac-498c-a686-889f791bd53d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:02.012372629Z\",\n      \"endTime\": \"2026-08-07T15:38:07.880331415Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"a68694cb26bd297d53889e4b8d624876fd75c4b8afd0235dc35b3bc657276b3f","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"44ec2ddbd6b01becc797252b99594cd00f8353c1d9a384e972ba4ab7b01d7ff2","test_name":"Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-7gg2l8\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ef07a0\u003e: \n    failed to create HCP cluster np-noedge-7gg2l8: failed waiting for cluster=\"np-noedge-7gg2l8\" in resourcegroup=\"rg-np-noedge-7gg2l8-7tgbtx6v2wzw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d9d0802-bf21-4d21-b75b-c540aec1feba\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d9d0802-bf21-4d21-b75b-c540aec1feba\",\n      \"name\": \"1d9d0802-bf21-4d21-b75b-c540aec1feba\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:24.519123762Z\",\n      \"endTime\": \"2026-08-07T15:58:26.855401935Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-7gg2l8\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ef07a0\u003e: \n    failed to create HCP cluster np-noedge-7gg2l8: failed waiting for cluster=\"np-noedge-7gg2l8\" in resourcegroup=\"rg-np-noedge-7gg2l8-7tgbtx6v2wzw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d9d0802-bf21-4d21-b75b-c540aec1feba\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d9d0802-bf21-4d21-b75b-c540aec1feba\",\n      \"name\": \"1d9d0802-bf21-4d21-b75b-c540aec1feba\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:24.519123762Z\",\n      \"endTime\": \"2026-08-07T15:58:26.855401935Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"b1720a318e8babfe874e69a92b53dfbb8f30ba6371afcd3152879c296ae9af1d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"021b2282da99c1c18e61ea792b9b369b1ab289c1956a25462f089fabe8b8f2d3","test_name":"Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a567e0\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-695gh2665l9l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35cec430-b27a-460d-ab8b-fa40f2f6e007\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35cec430-b27a-460d-ab8b-fa40f2f6e007\",\n      \"name\": \"35cec430-b27a-460d-ab8b-fa40f2f6e007\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:33.105687956Z\",\n      \"endTime\": \"2026-08-07T15:38:37.90302898Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a567e0\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-695gh2665l9l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35cec430-b27a-460d-ab8b-fa40f2f6e007\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/35cec430-b27a-460d-ab8b-fa40f2f6e007\",\n      \"name\": \"35cec430-b27a-460d-ab8b-fa40f2f6e007\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:33.105687956Z\",\n      \"endTime\": \"2026-08-07T15:38:37.90302898Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"b7408843e27af265c58bdf4b45c2a0e7ff2c82c174e540c35381e9f7e5287272","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"74749ba5ff7a90f44bdc6fae3690840173b458a11644b71c3594d04163b7673c","test_name":"Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e50ba0\u003e: \n    failed to create HCP cluster np-downgrade-8m4ss8: failed waiting for cluster=\"np-downgrade-8m4ss8\" in resourcegroup=\"rg-np-downgrade-8m4ss8-vvvrqstsfb9w\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/265886c9-a6ce-420d-9f4a-b1d32e37e426\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/265886c9-a6ce-420d-9f4a-b1d32e37e426\",\n      \"name\": \"265886c9-a6ce-420d-9f4a-b1d32e37e426\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:36.689848963Z\",\n      \"endTime\": \"2026-08-07T15:58:36.871323335Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e50ba0\u003e: \n    failed to create HCP cluster np-downgrade-8m4ss8: failed waiting for cluster=\"np-downgrade-8m4ss8\" in resourcegroup=\"rg-np-downgrade-8m4ss8-vvvrqstsfb9w\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/265886c9-a6ce-420d-9f4a-b1d32e37e426\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/265886c9-a6ce-420d-9f4a-b1d32e37e426\",\n      \"name\": \"265886c9-a6ce-420d-9f4a-b1d32e37e426\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:39:36.689848963Z\",\n      \"endTime\": \"2026-08-07T15:58:36.871323335Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"b9ddfb7fe216d0f945ffee84a75f973c406e2c2acf033f1911fa4bb6411d4b2d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"c5f121827dcc69a596d4c25871fdd5475afea6eb449844be2f893ab2d7744428","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-lfhxvtqv62md/cluster-candidate-4-20-xl4dng should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d8ac00\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-xl4dng: failed waiting for cluster=\"cluster-candidate-4-20-xl4dng\" in resourcegroup=\"rg-candidate-4-20-lfhxvtqv62md\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d560387b-98fa-400a-a8bc-5d44c7dbedfc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d560387b-98fa-400a-a8bc-5d44c7dbedfc\",\n      \"name\": \"d560387b-98fa-400a-a8bc-5d44c7dbedfc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:30.29873155Z\",\n      \"endTime\": \"2026-08-07T15:35:47.561593663Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-lfhxvtqv62md/cluster-candidate-4-20-xl4dng should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d8ac00\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-xl4dng: failed waiting for cluster=\"cluster-candidate-4-20-xl4dng\" in resourcegroup=\"rg-candidate-4-20-lfhxvtqv62md\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d560387b-98fa-400a-a8bc-5d44c7dbedfc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d560387b-98fa-400a-a8bc-5d44c7dbedfc\",\n      \"name\": \"d560387b-98fa-400a-a8bc-5d44c7dbedfc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:30.29873155Z\",\n      \"endTime\": \"2026-08-07T15:35:47.561593663Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"c0dd3b716fad621db1dc5bcb70286c1ecb2d2da956e6a5954a830e18b9aafd89","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"dfe5efdf662b8d6ad205f747617c8563e09fc6f03908cfc4dc7bdbd3ab1a7f28","test_name":"Customer should be able to create an HCP cluster with Image Registry not present","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00107b080\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-p5zk7sbz7vfj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370663d9-ca9e-42ca-b14e-b4044dfac98a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370663d9-ca9e-42ca-b14e-b4044dfac98a\",\n      \"name\": \"370663d9-ca9e-42ca-b14e-b4044dfac98a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:27.10370427Z\",\n      \"endTime\": \"2026-08-07T15:38:27.851206407Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00107b080\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-p5zk7sbz7vfj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370663d9-ca9e-42ca-b14e-b4044dfac98a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370663d9-ca9e-42ca-b14e-b4044dfac98a\",\n      \"name\": \"370663d9-ca9e-42ca-b14e-b4044dfac98a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:27.10370427Z\",\n      \"endTime\": \"2026-08-07T15:38:27.851206407Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"cfe383973a1cd4a4bd5f4e9d12c4b81f721e99506b5242b845dbd8f3da6e3871","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"9d6d196e442a9c431b085383d01a29883f369fcf77598556f5f91c887c194862","test_name":"Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001470360\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-slqw46l7544m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4bfb702-8c5c-486f-878f-e9acd0944e0c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4bfb702-8c5c-486f-878f-e9acd0944e0c\",\n      \"name\": \"b4bfb702-8c5c-486f-878f-e9acd0944e0c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:55:37.069958735Z\",\n      \"endTime\": \"2026-08-07T15:14:46.175487865Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001470360\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-slqw46l7544m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4bfb702-8c5c-486f-878f-e9acd0944e0c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b4bfb702-8c5c-486f-878f-e9acd0944e0c\",\n      \"name\": \"b4bfb702-8c5c-486f-878f-e9acd0944e0c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:55:37.069958735Z\",\n      \"endTime\": \"2026-08-07T15:14:46.175487865Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d57b02b35d721092270b6858cf02d17b1dfe023b6940385e617c54880769583a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"01215d187e7929f251a8d98f4941e267c94303e390aceb2d52d03dbb5c1b02d4","test_name":"Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-jm89zv\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001290980\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-jm89zv: failed waiting for cluster=\"cp-ystream-upgrade-4-21-jm89zv\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-m2z6w6flbncq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b10a5921-07ff-41b8-a752-24d73eb9729f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b10a5921-07ff-41b8-a752-24d73eb9729f\",\n      \"name\": \"b10a5921-07ff-41b8-a752-24d73eb9729f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:17:09.07729999Z\",\n      \"endTime\": \"2026-08-07T15:36:17.820229822Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-jm89zv\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001290980\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-jm89zv: failed waiting for cluster=\"cp-ystream-upgrade-4-21-jm89zv\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-m2z6w6flbncq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b10a5921-07ff-41b8-a752-24d73eb9729f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b10a5921-07ff-41b8-a752-24d73eb9729f\",\n      \"name\": \"b10a5921-07ff-41b8-a752-24d73eb9729f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:17:09.07729999Z\",\n      \"endTime\": \"2026-08-07T15:36:17.820229822Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d61f0d0d58c8e9fa45820ccdfa2c25cbec411f2c6e7dc7a63c03127fa7a77c39","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"57cec574ef011ddc63df19b276cb635f6cbf43e6e9fd68da8fb9411a4055bb08","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-fm7r8n with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e0efe0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-fm7r8n: failed waiting for cluster=\"np-version-upgrade-cluster-fm7r8n\" in resourcegroup=\"rg-np-version-upgrade-fm7r8n-724bzc7vvcj9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2bb675dd-03b7-4fdd-a1a2-61bac13c476f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2bb675dd-03b7-4fdd-a1a2-61bac13c476f\",\n      \"name\": \"2bb675dd-03b7-4fdd-a1a2-61bac13c476f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:38:51.454981357Z\",\n      \"endTime\": \"2026-08-07T15:57:56.809685658Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-fm7r8n with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e0efe0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-fm7r8n: failed waiting for cluster=\"np-version-upgrade-cluster-fm7r8n\" in resourcegroup=\"rg-np-version-upgrade-fm7r8n-724bzc7vvcj9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2bb675dd-03b7-4fdd-a1a2-61bac13c476f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2bb675dd-03b7-4fdd-a1a2-61bac13c476f\",\n      \"name\": \"2bb675dd-03b7-4fdd-a1a2-61bac13c476f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:38:51.454981357Z\",\n      \"endTime\": \"2026-08-07T15:57:56.809685658Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"dda6335c6cadb217161ea743ab53c5ae12ac09255af97a1d7eae29466f5e25fd","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"16605a62050506db8c803a6af41b021ea4f5391e8f5a52796294e1f71f2e184d","test_name":"Authorized CIDRs Connectivity should allow API access only from authorized VM IP","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001026dc0\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-4bczbl5pshts\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c8ec5b7-d5b6-4880-b26c-cdfdc4c603a0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c8ec5b7-d5b6-4880-b26c-cdfdc4c603a0\",\n      \"name\": \"6c8ec5b7-d5b6-4880-b26c-cdfdc4c603a0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:50.789743504Z\",\n      \"endTime\": \"2026-08-07T15:13:55.998584124Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001026dc0\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-4bczbl5pshts\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c8ec5b7-d5b6-4880-b26c-cdfdc4c603a0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c8ec5b7-d5b6-4880-b26c-cdfdc4c603a0\",\n      \"name\": \"6c8ec5b7-d5b6-4880-b26c-cdfdc4c603a0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:50.789743504Z\",\n      \"endTime\": \"2026-08-07T15:13:55.998584124Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"e3ed261ade2812c2b0572f99ded49c271e2a06ba1863e6bd9ac7609de94c8634","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"e326339ad9b3e9f4174e7533b5201f672a3f026fc49d1af623de0c5677817e56","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-6ssdzf\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f4a580\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-6ssdzf: failed waiting for cluster=\"cluster-zstream-4-19-6ssdzf\" in resourcegroup=\"rg-zstream-upgrade-4-19-7rq8pvtz666v\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/511e49f8-1f0e-446f-ae2e-96422a70f9c4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/511e49f8-1f0e-446f-ae2e-96422a70f9c4\",\n      \"name\": \"511e49f8-1f0e-446f-ae2e-96422a70f9c4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:55.971181939Z\",\n      \"endTime\": \"2026-08-07T15:35:57.688741756Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-6ssdzf\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f4a580\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-6ssdzf: failed waiting for cluster=\"cluster-zstream-4-19-6ssdzf\" in resourcegroup=\"rg-zstream-upgrade-4-19-7rq8pvtz666v\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/511e49f8-1f0e-446f-ae2e-96422a70f9c4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/511e49f8-1f0e-446f-ae2e-96422a70f9c4\",\n      \"name\": \"511e49f8-1f0e-446f-ae2e-96422a70f9c4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:16:55.971181939Z\",\n      \"endTime\": \"2026-08-07T15:35:57.688741756Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"e94eaba833548275af0002ee9484813e8165917ddf090074cd83257bdb206fa6","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"a34c57925c72454c6a48fd37184216056eb5535d84add4ab3b2e53d43c2f4f12","test_name":"Customer should be able to create an HCP cluster then delete it by deleting the customer resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a229a0\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-bs8c6mwtmwz4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ec246eec-8bfe-4b08-869b-2f0f1e2d8785\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ec246eec-8bfe-4b08-869b-2f0f1e2d8785\",\n      \"name\": \"ec246eec-8bfe-4b08-869b-2f0f1e2d8785\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:53:46.64766679Z\",\n      \"endTime\": \"2026-08-07T15:12:55.927527313Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a229a0\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-bs8c6mwtmwz4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ec246eec-8bfe-4b08-869b-2f0f1e2d8785\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ec246eec-8bfe-4b08-869b-2f0f1e2d8785\",\n      \"name\": \"ec246eec-8bfe-4b08-869b-2f0f1e2d8785\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:53:46.64766679Z\",\n      \"endTime\": \"2026-08-07T15:12:55.927527313Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ee117c34f6f92661a1fb22cb4bfe7f7c5dd0d76f5c395a39f10e7fbe439396ec","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"210a3d605e7b8e067ce0d297e3ad6db49ba695dfdb53bcb2768834ce7e0bdb0f","test_name":"Customer should be able to update node pool labels and taints","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010608c0\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-xcqrzgkj7knk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12f3228f-708f-4f8b-8a11-da90fb619746\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12f3228f-708f-4f8b-8a11-da90fb619746\",\n      \"name\": \"12f3228f-708f-4f8b-8a11-da90fb619746\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:20:08.095331782Z\",\n      \"endTime\": \"2026-08-07T15:39:17.892097154Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010608c0\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-xcqrzgkj7knk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12f3228f-708f-4f8b-8a11-da90fb619746\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12f3228f-708f-4f8b-8a11-da90fb619746\",\n      \"name\": \"12f3228f-708f-4f8b-8a11-da90fb619746\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:20:08.095331782Z\",\n      \"endTime\": \"2026-08-07T15:39:17.892097154Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ef3e296e5585a4e3dd9b4af12a17ec459810866f3b03618ebcdaf365c1222772","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"bbe5bc34d856b7fb86b2e954c41a5bc935dbcca3b6b209dadab57a318e8fa38c","test_name":"Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001008aa0\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-wjnvfvnxp547\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f56a40bb-d668-4c3b-91ac-815ee2773dfd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f56a40bb-d668-4c3b-91ac-815ee2773dfd\",\n      \"name\": \"f56a40bb-d668-4c3b-91ac-815ee2773dfd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:30.244030443Z\",\n      \"endTime\": \"2026-08-07T15:38:37.865021517Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001008aa0\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-wjnvfvnxp547\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f56a40bb-d668-4c3b-91ac-815ee2773dfd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f56a40bb-d668-4c3b-91ac-815ee2773dfd\",\n      \"name\": \"f56a40bb-d668-4c3b-91ac-815ee2773dfd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T15:19:30.244030443Z\",\n      \"endTime\": \"2026-08-07T15:38:37.865021517Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"f861276cfa9f4f67b20dd8cd916fd5ec9f0a1ad6c88d7f841a781547d30a78b9","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"d42cea7e099f8ef8c16bc0ce8a7766e75faceeaef522f8387c787103c0a1785a","test_name":"Customer should be able to create node pools with ARM64-based VMs","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009708a0\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-x5dh5x7lhq7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27cd9b4c-e2c4-4591-af28-91de8d626cb2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27cd9b4c-e2c4-4591-af28-91de8d626cb2\",\n      \"name\": \"27cd9b4c-e2c4-4591-af28-91de8d626cb2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:17.343249834Z\",\n      \"endTime\": \"2026-08-07T15:13:25.904584801Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009708a0\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-x5dh5x7lhq7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27cd9b4c-e2c4-4591-af28-91de8d626cb2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/27cd9b4c-e2c4-4591-af28-91de8d626cb2\",\n      \"name\": \"27cd9b4c-e2c4-4591-af28-91de8d626cb2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:17.343249834Z\",\n      \"endTime\": \"2026-08-07T15:13:25.904584801Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ffffbb77a2d037bb143094560f5f340d08e00c710e07720008f97ae958c199f2","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085735196925104128","occurred_at":"2026-08-07T14:20:01Z","failed_at":"e2e","signature_id":"20115f52aacc3454860fa2ea459ecbc73803b1beda7eccd723513c553395985d","test_name":"Customer should be able to create an HCP cluster and custom node pool osDisk size","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001796ca0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c8t6ksmrqwcx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/367b02a4-951f-4cf8-a1c9-ee3eaa5fb73f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/367b02a4-951f-4cf8-a1c9-ee3eaa5fb73f\",\n      \"name\": \"367b02a4-951f-4cf8-a1c9-ee3eaa5fb73f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:04.145303699Z\",\n      \"endTime\": \"2026-08-07T15:13:06.333061764Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001796ca0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c8t6ksmrqwcx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/367b02a4-951f-4cf8-a1c9-ee3eaa5fb73f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/367b02a4-951f-4cf8-a1c9-ee3eaa5fb73f\",\n      \"name\": \"367b02a4-951f-4cf8-a1c9-ee3eaa5fb73f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T14:54:04.145303699Z\",\n      \"endTime\": \"2026-08-07T15:13:06.333061764Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}}],"failure_pattern_summary":{"signature_count":45,"failure_pattern_ids":["0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f","e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b"],"matched_occurrences":45,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T11:55:52Z"},"failed_at":["e2e"],"failed_test_count":44,"occurrences":[{"row_id":"0438f2be20a25a6905114a084eb29b7d43d66f25a68c0c9e883efc2bf5665819","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"14c80964b9874c9081110be783c8e717ab9db60e5a66ec45327b015be13e84aa","test_name":"Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed\nfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed","failure_pattern_match":{"status":"clustered","failure_pattern_id":"e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b","failure_pattern":"Cluster provisioning failed","search_query":"Cluster provisioning failed"}},{"row_id":"08a0465c25239d76809038cd084d10515162edf115dad96ca9a1a9d6f60e9055","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"17b9fa462ad3293ae5c44dbb173ffe54a3206df5f7bb43f2726db0017de95e70","test_name":"Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008813a0\u003e: \n    failed to create HCP cluster np-downgrade-qgnx8g: failed waiting for cluster=\"np-downgrade-qgnx8g\" in resourcegroup=\"rg-np-downgrade-qgnx8g-l6jvxwpd2mzx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/644d9079-ed6f-4b1f-ab2e-4725b9c863f6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/644d9079-ed6f-4b1f-ab2e-4725b9c863f6\",\n      \"name\": \"644d9079-ed6f-4b1f-ab2e-4725b9c863f6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:30.527043352Z\",\n      \"endTime\": \"2026-08-07T13:19:30.660300158Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008813a0\u003e: \n    failed to create HCP cluster np-downgrade-qgnx8g: failed waiting for cluster=\"np-downgrade-qgnx8g\" in resourcegroup=\"rg-np-downgrade-qgnx8g-l6jvxwpd2mzx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/644d9079-ed6f-4b1f-ab2e-4725b9c863f6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/644d9079-ed6f-4b1f-ab2e-4725b9c863f6\",\n      \"name\": \"644d9079-ed6f-4b1f-ab2e-4725b9c863f6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:30.527043352Z\",\n      \"endTime\": \"2026-08-07T13:19:30.660300158Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"28b71790175175ca6d28a8c8585026ad5c208ed85c89573fca22303c794510db","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"c23098826952b03b03b5f37e2dd466af45f541c2c64d2b9693af98e64f8db4a5","test_name":"Customer should be able to create an HCP cluster and custom node pool osDisk size","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00152efc0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fmg4hcn62t7t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d0403be-cf4c-4ca0-bea2-bbef12dfd502\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d0403be-cf4c-4ca0-bea2-bbef12dfd502\",\n      \"name\": \"4d0403be-cf4c-4ca0-bea2-bbef12dfd502\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:43.229408811Z\",\n      \"endTime\": \"2026-08-07T12:33:49.589577673Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00152efc0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fmg4hcn62t7t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d0403be-cf4c-4ca0-bea2-bbef12dfd502\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d0403be-cf4c-4ca0-bea2-bbef12dfd502\",\n      \"name\": \"4d0403be-cf4c-4ca0-bea2-bbef12dfd502\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:43.229408811Z\",\n      \"endTime\": \"2026-08-07T12:33:49.589577673Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"294c69f2f074c6ca7674daebfc279ba1c77ec11608caca0a0b5de9637a3de229","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"bb534518404be8f2f26768706d98c6912cc0e62e2d2cc2515e4c682a3c06e027","test_name":"Customer should be able to use workload identity via the cluster OIDC issuer URL","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e97000\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7fkqlc4qsr92\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5143f7f3-5a5f-4b77-93dc-40a45c1e102f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5143f7f3-5a5f-4b77-93dc-40a45c1e102f\",\n      \"name\": \"5143f7f3-5a5f-4b77-93dc-40a45c1e102f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:01:14.88444793Z\",\n      \"endTime\": \"2026-08-07T13:20:20.621983431Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e97000\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7fkqlc4qsr92\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5143f7f3-5a5f-4b77-93dc-40a45c1e102f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5143f7f3-5a5f-4b77-93dc-40a45c1e102f\",\n      \"name\": \"5143f7f3-5a5f-4b77-93dc-40a45c1e102f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:01:14.88444793Z\",\n      \"endTime\": \"2026-08-07T13:20:20.621983431Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"298d99b71a6ebbbd43e51ae1f5f6eca46f4162e5a8f5b40f1b4ead18056ee292","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"c809b9341de13909a91559e68c1dcb2833ef4e3acbe48e1036a91af323ad76da","test_name":"Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ef6940\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-jrzb2c95htvf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/11209a01-df88-4c5a-97a8-363e0267a08e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/11209a01-df88-4c5a-97a8-363e0267a08e\",\n      \"name\": \"11209a01-df88-4c5a-97a8-363e0267a08e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:16:15.107494855Z\",\n      \"endTime\": \"2026-08-07T12:35:19.537299803Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ef6940\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-jrzb2c95htvf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/11209a01-df88-4c5a-97a8-363e0267a08e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/11209a01-df88-4c5a-97a8-363e0267a08e\",\n      \"name\": \"11209a01-df88-4c5a-97a8-363e0267a08e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:16:15.107494855Z\",\n      \"endTime\": \"2026-08-07T12:35:19.537299803Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"2bb62fa9b860f49be818094b9f72274a704f4c35591c08fff1ed16e621c0e1bf","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"8aed6fcafda8e1a1783469b44ea2e98e2307b7c8def80a48403e4c8fbaed1f1a","test_name":"Customer should respect cluster-wide node limits with nodepool autoscaling","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010ba940\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tkrsb5842rqz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5344efa-8734-4cbd-b510-27491b044dfe\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5344efa-8734-4cbd-b510-27491b044dfe\",\n      \"name\": \"b5344efa-8734-4cbd-b510-27491b044dfe\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:07.461308507Z\",\n      \"endTime\": \"2026-08-07T12:59:09.224573361Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010ba940\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tkrsb5842rqz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5344efa-8734-4cbd-b510-27491b044dfe\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5344efa-8734-4cbd-b510-27491b044dfe\",\n      \"name\": \"b5344efa-8734-4cbd-b510-27491b044dfe\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:07.461308507Z\",\n      \"endTime\": \"2026-08-07T12:59:09.224573361Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"3f6e5cb99d9adadb6d85bdf73ef538f9cfb21690bbef8a86a3cfa7f42b4a051c","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"c7c48f1285fcb40e3c337a08b89c292d11a1fe19ebcd357a806359230ffeac3d","test_name":"Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-gqqc89 at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000fd0160\u003e: \n    failed to create HCP cluster np-skip-minor-gqqc89: failed waiting for cluster=\"np-skip-minor-gqqc89\" in resourcegroup=\"rg-np-skip-minor-gqqc89-d478g24m4l7q\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d73efe4f-bd71-4106-ad03-393cbdd63a0e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d73efe4f-bd71-4106-ad03-393cbdd63a0e\",\n      \"name\": \"d73efe4f-bd71-4106-ad03-393cbdd63a0e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:40.32922575Z\",\n      \"endTime\": \"2026-08-07T13:19:40.634355282Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-gqqc89 at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000fd0160\u003e: \n    failed to create HCP cluster np-skip-minor-gqqc89: failed waiting for cluster=\"np-skip-minor-gqqc89\" in resourcegroup=\"rg-np-skip-minor-gqqc89-d478g24m4l7q\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d73efe4f-bd71-4106-ad03-393cbdd63a0e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d73efe4f-bd71-4106-ad03-393cbdd63a0e\",\n      \"name\": \"d73efe4f-bd71-4106-ad03-393cbdd63a0e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:40.32922575Z\",\n      \"endTime\": \"2026-08-07T13:19:40.634355282Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"3ffb6b5d5c7b5ec5ff8702d8fd912aab0e89d9f621d284bf72bad1ba80699095","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"3acf4a36e898bcd4be4cbf53c61be31b047cd11a76f013065dc60a1bff5fa765","test_name":"Customer should be able to create an HCP cluster with Image Registry not present","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f08920\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zk2mzt57xg5j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cb5b70e-7045-47ab-b9f7-bcf911cfe87c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cb5b70e-7045-47ab-b9f7-bcf911cfe87c\",\n      \"name\": \"6cb5b70e-7045-47ab-b9f7-bcf911cfe87c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:07.164341393Z\",\n      \"endTime\": \"2026-08-07T12:59:09.182930028Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f08920\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zk2mzt57xg5j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cb5b70e-7045-47ab-b9f7-bcf911cfe87c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cb5b70e-7045-47ab-b9f7-bcf911cfe87c\",\n      \"name\": \"6cb5b70e-7045-47ab-b9f7-bcf911cfe87c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:07.164341393Z\",\n      \"endTime\": \"2026-08-07T12:59:09.182930028Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4028a1850fe9fd8faa4a8049c7f1c3a51c0fb71d2facbab85f168dc76d4d6037","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"3db87cf0739292d3c09e09b1ebf945aa516706e25aa3a8e41f1289a7de8fd8a4","test_name":"Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d395e0\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-khv7rgn9bftt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddbdae56-0f17-45d0-9b50-3f529b8b51a2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddbdae56-0f17-45d0-9b50-3f529b8b51a2\",\n      \"name\": \"ddbdae56-0f17-45d0-9b50-3f529b8b51a2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:15:49.353517484Z\",\n      \"endTime\": \"2026-08-07T12:34:49.557216916Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d395e0\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-khv7rgn9bftt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddbdae56-0f17-45d0-9b50-3f529b8b51a2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddbdae56-0f17-45d0-9b50-3f529b8b51a2\",\n      \"name\": \"ddbdae56-0f17-45d0-9b50-3f529b8b51a2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:15:49.353517484Z\",\n      \"endTime\": \"2026-08-07T12:34:49.557216916Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"47ccbb5a25631924c0a7f2f59e6b96014d01fc766dfa7a5c74a8c518e1f34ac5","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"f8fcd832392b100024ef561f2885f2ecde46c911fd0bfa5794ac6d1326581a01","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-96dhzz\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00168a1a0\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-96dhzz: failed waiting for cluster=\"cluster-zstream-4-22-96dhzz\" in resourcegroup=\"rg-zstream-upgrade-4-22-vmd2wx2d4f8v\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77aa7c80-d2f4-4776-9e8e-0cdf57d8e7f4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77aa7c80-d2f4-4776-9e8e-0cdf57d8e7f4\",\n      \"name\": \"77aa7c80-d2f4-4776-9e8e-0cdf57d8e7f4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:01.49295909Z\",\n      \"endTime\": \"2026-08-07T12:57:09.102280408Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-96dhzz\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00168a1a0\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-96dhzz: failed waiting for cluster=\"cluster-zstream-4-22-96dhzz\" in resourcegroup=\"rg-zstream-upgrade-4-22-vmd2wx2d4f8v\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77aa7c80-d2f4-4776-9e8e-0cdf57d8e7f4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/77aa7c80-d2f4-4776-9e8e-0cdf57d8e7f4\",\n      \"name\": \"77aa7c80-d2f4-4776-9e8e-0cdf57d8e7f4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:01.49295909Z\",\n      \"endTime\": \"2026-08-07T12:57:09.102280408Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"5063382d21f6dc407533bdbdad1cf04507010454deb53cfe6cd8c7026eddfb51","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"a06b8efdd4dfd86e0cbb859a50f1335f152f7a71af525911330bb70623a1670d","test_name":"Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-gphzpw with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f02ae0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-gphzpw: failed waiting for cluster=\"np-version-upgrade-cluster-gphzpw\" in resourcegroup=\"rg-np-version-upgrade-gphzpw-cvgrn8w28ts9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ece9037d-87c6-4c68-931c-3a6e8310c4e6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ece9037d-87c6-4c68-931c-3a6e8310c4e6\",\n      \"name\": \"ece9037d-87c6-4c68-931c-3a6e8310c4e6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:29.520922716Z\",\n      \"endTime\": \"2026-08-07T13:19:30.624596769Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-gphzpw with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f02ae0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-gphzpw: failed waiting for cluster=\"np-version-upgrade-cluster-gphzpw\" in resourcegroup=\"rg-np-version-upgrade-gphzpw-cvgrn8w28ts9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ece9037d-87c6-4c68-931c-3a6e8310c4e6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ece9037d-87c6-4c68-931c-3a6e8310c4e6\",\n      \"name\": \"ece9037d-87c6-4c68-931c-3a6e8310c4e6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:29.520922716Z\",\n      \"endTime\": \"2026-08-07T13:19:30.624596769Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"607d20fd4ae73b43325ecaee19da6708b556a78b4ed85159ec42add147a17e90","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"5247dc2e35616608fe3df5970a38341bb30864bf640f9a7f5629ee9db3724657","test_name":"Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008e4e20\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-njn4sztwtwcs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4367d41f-2bf1-4d5e-8825-def36a4c6aa9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4367d41f-2bf1-4d5e-8825-def36a4c6aa9\",\n      \"name\": \"4367d41f-2bf1-4d5e-8825-def36a4c6aa9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:32.094368965Z\",\n      \"endTime\": \"2026-08-07T12:33:39.679750193Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008e4e20\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-njn4sztwtwcs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4367d41f-2bf1-4d5e-8825-def36a4c6aa9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4367d41f-2bf1-4d5e-8825-def36a4c6aa9\",\n      \"name\": \"4367d41f-2bf1-4d5e-8825-def36a4c6aa9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:32.094368965Z\",\n      \"endTime\": \"2026-08-07T12:33:39.679750193Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"60c0253267088fd18c74e0246776154d03c2916b561c31c03e6052604a379d1a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"238907ba56efa07df54e4b5f84003e9a5ad35a13eabbf5b14adfb45cdd6cf62e","test_name":"Customer should not be able to reuse subnets and NSGs between clusters","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000bc0f40\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-d2knvphdj9bj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e5fa84bc-affb-44f4-9444-d03d25c36f24\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e5fa84bc-affb-44f4-9444-d03d25c36f24\",\n      \"name\": \"e5fa84bc-affb-44f4-9444-d03d25c36f24\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:42.246199863Z\",\n      \"endTime\": \"2026-08-07T12:33:49.671114842Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000bc0f40\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-d2knvphdj9bj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e5fa84bc-affb-44f4-9444-d03d25c36f24\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e5fa84bc-affb-44f4-9444-d03d25c36f24\",\n      \"name\": \"e5fa84bc-affb-44f4-9444-d03d25c36f24\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:42.246199863Z\",\n      \"endTime\": \"2026-08-07T12:33:49.671114842Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"624f2f0cd22b8ec9bc96a57a287df57f8955b91e0a8d98657e6435586636d447","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"d205941f6cdbe8757b76859e859624d6182159c7e8e927af54f6cc3d9c9007a1","test_name":"Customer should be able to create an HCP cluster with back-level version 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:150]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc001576990\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rbdxzzdg8pgg\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-rbdxzzdg8pgg, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:150]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc001576990\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rbdxzzdg8pgg\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-rbdxzzdg8pgg, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-br948jjgf9df\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-br948jjgf9df, error: context deadline exceeded"}},{"row_id":"634019c74fe08cc34f9813d0cb2707c30da1ecbd08a50671d9b37cd72115b6bd","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"0ab976e378e0a998708c0b29776cf3ad86523734d343888bb96d6f572717ec4b","test_name":"Customer should create an HCP cluster and validate TLS certificates","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008bc340\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bnxvbq5c6mnj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6897e8cd-5484-4ab3-9d7e-e7394a55d49b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6897e8cd-5484-4ab3-9d7e-e7394a55d49b\",\n      \"name\": \"6897e8cd-5484-4ab3-9d7e-e7394a55d49b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:45.179107272Z\",\n      \"endTime\": \"2026-08-07T12:33:49.557406995Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008bc340\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-bnxvbq5c6mnj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6897e8cd-5484-4ab3-9d7e-e7394a55d49b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6897e8cd-5484-4ab3-9d7e-e7394a55d49b\",\n      \"name\": \"6897e8cd-5484-4ab3-9d7e-e7394a55d49b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:45.179107272Z\",\n      \"endTime\": \"2026-08-07T12:33:49.557406995Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"676b95a737caf7858096bf96271c26f4803347d86382e9c93d374670486a6983","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"fe000511c3eefa2fc5073b836d8b11582f257ade46c054a434ce50395bf206b8","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-d97tfc9k4slk/cluster-candidate-4-21-68m5vt should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00049e720\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-68m5vt: failed waiting for cluster=\"cluster-candidate-4-21-68m5vt\" in resourcegroup=\"rg-candidate-4-21-d97tfc9k4slk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/16d2c67b-2b5f-46a9-b3eb-ff6c7f9a4f60\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/16d2c67b-2b5f-46a9-b3eb-ff6c7f9a4f60\",\n      \"name\": \"16d2c67b-2b5f-46a9-b3eb-ff6c7f9a4f60\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:10.020742072Z\",\n      \"endTime\": \"2026-08-07T12:56:19.197946564Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-d97tfc9k4slk/cluster-candidate-4-21-68m5vt should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00049e720\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-68m5vt: failed waiting for cluster=\"cluster-candidate-4-21-68m5vt\" in resourcegroup=\"rg-candidate-4-21-d97tfc9k4slk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/16d2c67b-2b5f-46a9-b3eb-ff6c7f9a4f60\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/16d2c67b-2b5f-46a9-b3eb-ff6c7f9a4f60\",\n      \"name\": \"16d2c67b-2b5f-46a9-b3eb-ff6c7f9a4f60\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:10.020742072Z\",\n      \"endTime\": \"2026-08-07T12:56:19.197946564Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"677e65af0940ff20c50d606b6d1c8ba4d06dec728c18cacb304308cb460f88d1","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"b7ab86b3f0bbe5fe176329f38e2f9cb5a42e5be7904749505f8b016e622a8e8d","test_name":"Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e929c0\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-k2sljkw5dsmm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/893230a6-82b7-40a3-8d7d-8aba9a855e58\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/893230a6-82b7-40a3-8d7d-8aba9a855e58\",\n      \"name\": \"893230a6-82b7-40a3-8d7d-8aba9a855e58\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:17:31.599754615Z\",\n      \"endTime\": \"2026-08-07T12:36:39.535971824Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e929c0\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-k2sljkw5dsmm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/893230a6-82b7-40a3-8d7d-8aba9a855e58\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/893230a6-82b7-40a3-8d7d-8aba9a855e58\",\n      \"name\": \"893230a6-82b7-40a3-8d7d-8aba9a855e58\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:17:31.599754615Z\",\n      \"endTime\": \"2026-08-07T12:36:39.535971824Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"6fa2087e755e40b232ddfb6ba3dbfa90bde182cee315fea8759700f8dbf3972b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"de2acacd562f001707e7eb3d33ccee4721fcad47ea6613293a81cfaaeabab3d7","test_name":"Customer should be able to create a cluster with an external auth config and get the external auth config","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f12620\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25wffkhglctg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f277d92a-ca70-4469-ac5b-ffbd81677d70\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f277d92a-ca70-4469-ac5b-ffbd81677d70\",\n      \"name\": \"f277d92a-ca70-4469-ac5b-ffbd81677d70\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:01.922892065Z\",\n      \"endTime\": \"2026-08-07T12:57:09.048998452Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f12620\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25wffkhglctg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f277d92a-ca70-4469-ac5b-ffbd81677d70\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f277d92a-ca70-4469-ac5b-ffbd81677d70\",\n      \"name\": \"f277d92a-ca70-4469-ac5b-ffbd81677d70\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:01.922892065Z\",\n      \"endTime\": \"2026-08-07T12:57:09.048998452Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"76dae7cd25b257c9bc4e6c58d20e607d2ec6687c5b0a9e7f5ab5b22854041141","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"d0ba60935a9160ce7fba012d47794a2a7c9987de7d0044e97c4f9feeae3dbb41","test_name":"Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00116f160\u003e: \n    failed to create HCP cluster np-dg-minor-z77scs: failed waiting for cluster=\"np-dg-minor-z77scs\" in resourcegroup=\"rg-np-dg-minor-z77scs-zvxbf9z7drfx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73b896f0-5460-449d-bff8-d734b41e4574\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73b896f0-5460-449d-bff8-d734b41e4574\",\n      \"name\": \"73b896f0-5460-449d-bff8-d734b41e4574\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:01:05.562866558Z\",\n      \"endTime\": \"2026-08-07T13:20:10.62761353Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00116f160\u003e: \n    failed to create HCP cluster np-dg-minor-z77scs: failed waiting for cluster=\"np-dg-minor-z77scs\" in resourcegroup=\"rg-np-dg-minor-z77scs-zvxbf9z7drfx\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73b896f0-5460-449d-bff8-d734b41e4574\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73b896f0-5460-449d-bff8-d734b41e4574\",\n      \"name\": \"73b896f0-5460-449d-bff8-d734b41e4574\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:01:05.562866558Z\",\n      \"endTime\": \"2026-08-07T13:20:10.62761353Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"7e3ad0e8fe640ac3601dbdc5bca616329ae5235c98969e4dad01436a91e553f3","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"ddc5b3ba3a98c300e6cda39af2ad751b69425cd3cd655712ab11e9c7775397d8","test_name":"Customer should be able to update node pool labels and taints","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00090cec0\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-ldzjgxbvvj4p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6365cac9-1321-4718-bfdd-212fb9f40e0f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6365cac9-1321-4718-bfdd-212fb9f40e0f\",\n      \"name\": \"6365cac9-1321-4718-bfdd-212fb9f40e0f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:41:03.221203578Z\",\n      \"endTime\": \"2026-08-07T13:00:09.207155371Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00090cec0\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-ldzjgxbvvj4p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6365cac9-1321-4718-bfdd-212fb9f40e0f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6365cac9-1321-4718-bfdd-212fb9f40e0f\",\n      \"name\": \"6365cac9-1321-4718-bfdd-212fb9f40e0f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:41:03.221203578Z\",\n      \"endTime\": \"2026-08-07T13:00:09.207155371Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"82a426c41e303abbd5975f6a84d55ae998f2a111249c9a877a9f11fba0c0ac08","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"407af579501a67725089a8f5ce42f9e363a18ac40009ccdd3921e6eaf60cd792","test_name":"Authorized CIDRs Connectivity should allow API access only from authorized VM IP","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000de2fa0\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-hjtxzx96pp2k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d140c214-a1ad-4271-867f-264e90b96455\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d140c214-a1ad-4271-867f-264e90b96455\",\n      \"name\": \"d140c214-a1ad-4271-867f-264e90b96455\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:15:27.302091166Z\",\n      \"endTime\": \"2026-08-07T12:34:29.936084315Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000de2fa0\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-hjtxzx96pp2k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d140c214-a1ad-4271-867f-264e90b96455\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d140c214-a1ad-4271-867f-264e90b96455\",\n      \"name\": \"d140c214-a1ad-4271-867f-264e90b96455\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:15:27.302091166Z\",\n      \"endTime\": \"2026-08-07T12:34:29.936084315Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"867201dc28eb6a33e08f49bd45999b739e0ac42969176fd309a741e999268bfd","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"12b2310fd8469a17afcf2b1ef1b2732a37411a22dfba6c80eb38fcd508287c34","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-2x4wxl with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d805c0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-2x4wxl: failed waiting for cluster=\"np-version-upgrade-cluster-2x4wxl\" in resourcegroup=\"rg-np-version-upgrade-2x4wxl-f4tvm7hl6b4m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e323816-d547-4924-bb57-b2cc15e58f28\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e323816-d547-4924-bb57-b2cc15e58f28\",\n      \"name\": \"5e323816-d547-4924-bb57-b2cc15e58f28\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:29.900631828Z\",\n      \"endTime\": \"2026-08-07T13:19:30.648470988Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-2x4wxl with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d805c0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-2x4wxl: failed waiting for cluster=\"np-version-upgrade-cluster-2x4wxl\" in resourcegroup=\"rg-np-version-upgrade-2x4wxl-f4tvm7hl6b4m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e323816-d547-4924-bb57-b2cc15e58f28\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e323816-d547-4924-bb57-b2cc15e58f28\",\n      \"name\": \"5e323816-d547-4924-bb57-b2cc15e58f28\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:29.900631828Z\",\n      \"endTime\": \"2026-08-07T13:19:30.648470988Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"8d398f0bc1f95449537f1b1cad316e1697815b324c4a4ebd9b3ee40e96cbc90a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"c0835e29acc541f0839c8ba8be9a88ccb1e74d1a7af86d3f620cd4280efe4f9d","test_name":"HCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f58c80\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-hcl4s7btnvmm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26f1f2dd-e29e-40e5-aa7a-15952aa36038\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26f1f2dd-e29e-40e5-aa7a-15952aa36038\",\n      \"name\": \"26f1f2dd-e29e-40e5-aa7a-15952aa36038\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:30.603697081Z\",\n      \"endTime\": \"2026-08-07T12:57:39.198008756Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f58c80\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-hcl4s7btnvmm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26f1f2dd-e29e-40e5-aa7a-15952aa36038\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/26f1f2dd-e29e-40e5-aa7a-15952aa36038\",\n      \"name\": \"26f1f2dd-e29e-40e5-aa7a-15952aa36038\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:30.603697081Z\",\n      \"endTime\": \"2026-08-07T12:57:39.198008756Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"8dc5260db8b92b9cc46b84e0573b75e9158f171e38f1a4da9f7a41518e159615","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"485ee8e83627b7b601de2f6d3e63c02e297fc2fc9ca22b045fa69caa92b6bd2b","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-dtnkzd\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010b8bc0\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-dtnkzd: failed waiting for cluster=\"cluster-zstream-4-20-dtnkzd\" in resourcegroup=\"rg-zstream-upgrade-4-20-47nz7m9ct47p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e43511b-bcf2-43ed-aec6-cb4914d64dfa\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e43511b-bcf2-43ed-aec6-cb4914d64dfa\",\n      \"name\": \"3e43511b-bcf2-43ed-aec6-cb4914d64dfa\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:36.478836962Z\",\n      \"endTime\": \"2026-08-07T12:56:39.335938483Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-dtnkzd\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010b8bc0\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-dtnkzd: failed waiting for cluster=\"cluster-zstream-4-20-dtnkzd\" in resourcegroup=\"rg-zstream-upgrade-4-20-47nz7m9ct47p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e43511b-bcf2-43ed-aec6-cb4914d64dfa\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e43511b-bcf2-43ed-aec6-cb4914d64dfa\",\n      \"name\": \"3e43511b-bcf2-43ed-aec6-cb4914d64dfa\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:36.478836962Z\",\n      \"endTime\": \"2026-08-07T12:56:39.335938483Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"90451f2b0e585704f27fc55cef76119e8cb642220a8b80eb3f0234ce7452ce85","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"1a8a873a9e44711b81e59a2b4721d491189498778dea5a5b257323279d03ae46","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-n6hql8md5d2l/cluster-candidate-4-20-tnz4fv should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a1c1c0\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-tnz4fv: failed waiting for cluster=\"cluster-candidate-4-20-tnz4fv\" in resourcegroup=\"rg-candidate-4-20-n6hql8md5d2l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d390ad70-c154-4ae1-8d52-a1e4796e88e4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d390ad70-c154-4ae1-8d52-a1e4796e88e4\",\n      \"name\": \"d390ad70-c154-4ae1-8d52-a1e4796e88e4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:12.050990646Z\",\n      \"endTime\": \"2026-08-07T12:56:19.209833443Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-n6hql8md5d2l/cluster-candidate-4-20-tnz4fv should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a1c1c0\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-tnz4fv: failed waiting for cluster=\"cluster-candidate-4-20-tnz4fv\" in resourcegroup=\"rg-candidate-4-20-n6hql8md5d2l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d390ad70-c154-4ae1-8d52-a1e4796e88e4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d390ad70-c154-4ae1-8d52-a1e4796e88e4\",\n      \"name\": \"d390ad70-c154-4ae1-8d52-a1e4796e88e4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:12.050990646Z\",\n      \"endTime\": \"2026-08-07T12:56:19.209833443Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"9dde4165500a1a6d4d3ae68577f0402c5cf993e5271d7c60dc918d817f744d2f","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"d300aaf435d4e1646c82c19197d22a15dfba72adfeb68d3e388f472884465708","test_name":"Customer should be able to create an HCP cluster and manage pull secrets","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001081120\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-6z96vkgcrv8j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0c12142-7466-4522-9938-73bd3679eddc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0c12142-7466-4522-9938-73bd3679eddc\",\n      \"name\": \"b0c12142-7466-4522-9938-73bd3679eddc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:43.396367222Z\",\n      \"endTime\": \"2026-08-07T12:33:49.545193271Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001081120\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-6z96vkgcrv8j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0c12142-7466-4522-9938-73bd3679eddc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b0c12142-7466-4522-9938-73bd3679eddc\",\n      \"name\": \"b0c12142-7466-4522-9938-73bd3679eddc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:43.396367222Z\",\n      \"endTime\": \"2026-08-07T12:33:49.545193271Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"a16dac10a87c39f3ef6a095517e5f3d35a12500a5d039d6f7d6ee6895784650a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"27be310ea3dd90c5a4b73bede520b33d51c0da484b426a10729ae2387c745e51","test_name":"Customer should be able to create a HCP cluster and use cilium CNI plugin","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0015d66a0\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-6qqcnkbr26fb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8751752f-982c-4be4-91ae-f447cedbbcf5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8751752f-982c-4be4-91ae-f447cedbbcf5\",\n      \"name\": \"8751752f-982c-4be4-91ae-f447cedbbcf5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:42.513191974Z\",\n      \"endTime\": \"2026-08-07T12:33:49.588421908Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0015d66a0\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-6qqcnkbr26fb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8751752f-982c-4be4-91ae-f447cedbbcf5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8751752f-982c-4be4-91ae-f447cedbbcf5\",\n      \"name\": \"8751752f-982c-4be4-91ae-f447cedbbcf5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:42.513191974Z\",\n      \"endTime\": \"2026-08-07T12:33:49.588421908Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"b29e7221d4669d9b2491caae6a98aaa6c931b4f1b5de6fc2f2a4798fd870b11c","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"53823c99ae2e0997d7cf8d7ab6b0e8a8ab54f7dd0f8ec35070a98c19dbbe1c17","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-ll6jjd\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a121e0\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-ll6jjd: failed waiting for cluster=\"cluster-zstream-4-19-ll6jjd\" in resourcegroup=\"rg-zstream-upgrade-4-19-8fmr75zdwtr9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1aacfab9-0fc3-4fbf-86d1-77838cb8e1ec\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1aacfab9-0fc3-4fbf-86d1-77838cb8e1ec\",\n      \"name\": \"1aacfab9-0fc3-4fbf-86d1-77838cb8e1ec\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:00.927332064Z\",\n      \"endTime\": \"2026-08-07T12:57:08.962414067Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-ll6jjd\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a121e0\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-ll6jjd: failed waiting for cluster=\"cluster-zstream-4-19-ll6jjd\" in resourcegroup=\"rg-zstream-upgrade-4-19-8fmr75zdwtr9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1aacfab9-0fc3-4fbf-86d1-77838cb8e1ec\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1aacfab9-0fc3-4fbf-86d1-77838cb8e1ec\",\n      \"name\": \"1aacfab9-0fc3-4fbf-86d1-77838cb8e1ec\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:00.927332064Z\",\n      \"endTime\": \"2026-08-07T12:57:08.962414067Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"babc4a43b6aa86f39e4f5ca60460147187c7eb2e7f85fba00790b392593ca917","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"e6e2ee61a01dd991ee8fbde90a48d2215d91744e594caaeee717a465c007c975","test_name":"Customer should be able to create node pools with ARM64-based VMs","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005999e0\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-zjbrhd9lmk24\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7704622c-5cd7-467e-ab53-be27bf1dca9a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7704622c-5cd7-467e-ab53-be27bf1dca9a\",\n      \"name\": \"7704622c-5cd7-467e-ab53-be27bf1dca9a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:52.168971589Z\",\n      \"endTime\": \"2026-08-07T12:33:59.7867877Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005999e0\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-zjbrhd9lmk24\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7704622c-5cd7-467e-ab53-be27bf1dca9a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7704622c-5cd7-467e-ab53-be27bf1dca9a\",\n      \"name\": \"7704622c-5cd7-467e-ab53-be27bf1dca9a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:52.168971589Z\",\n      \"endTime\": \"2026-08-07T12:33:59.7867877Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"c0ad61432af50f0dd741ac10e4d089fc3ab0a6dd3fd09842b5caa2ae5cec3e30","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"9013e675989bd3312d95cfd426e5318859765876455d6850dc98ee4d503feff7","test_name":"ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.000s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.000s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","failure_pattern":"cluster entered terminal Failed state after role assignment deployment","search_query":"cluster entered terminal Failed state after role assignment deployment"}},{"row_id":"c0b640611ef7b3964238d5b8d563224a1e45eda7b382370b4dad521b91f33357","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"60cba38b24ca4884110b993cd854b1d99bd0ed7d1be861a2c852361386ef09d4","test_name":"Customer should be able to create an HCP cluster then delete it by deleting the customer resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00090b940\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-blqv948bt44h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e1df755-6f1f-4c87-8997-914cc1778dd8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e1df755-6f1f-4c87-8997-914cc1778dd8\",\n      \"name\": \"3e1df755-6f1f-4c87-8997-914cc1778dd8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:43.299678261Z\",\n      \"endTime\": \"2026-08-07T12:33:49.648235123Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00090b940\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-blqv948bt44h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e1df755-6f1f-4c87-8997-914cc1778dd8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3e1df755-6f1f-4c87-8997-914cc1778dd8\",\n      \"name\": \"3e1df755-6f1f-4c87-8997-914cc1778dd8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:14:43.299678261Z\",\n      \"endTime\": \"2026-08-07T12:33:49.648235123Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"c131f6aa931e5e877778e6a78f3dfb5144f5fc5e8309eab1b8ee042c45799033","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"bf43d1d48e1ae4b8bb38b7c882bf5bf623dd99e105edf3243c13d224ba870e0c","test_name":"Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-t585rc\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b088a0\u003e: \n    failed to create HCP cluster np-noedge-t585rc: failed waiting for cluster=\"np-noedge-t585rc\" in resourcegroup=\"rg-np-noedge-t585rc-czb8rx8q8gqw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/15bc30b9-fad7-4ef1-9263-ef766496d2c6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/15bc30b9-fad7-4ef1-9263-ef766496d2c6\",\n      \"name\": \"15bc30b9-fad7-4ef1-9263-ef766496d2c6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:40.843091586Z\",\n      \"endTime\": \"2026-08-07T13:19:40.666609145Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-t585rc\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b088a0\u003e: \n    failed to create HCP cluster np-noedge-t585rc: failed waiting for cluster=\"np-noedge-t585rc\" in resourcegroup=\"rg-np-noedge-t585rc-czb8rx8q8gqw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/15bc30b9-fad7-4ef1-9263-ef766496d2c6\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/15bc30b9-fad7-4ef1-9263-ef766496d2c6\",\n      \"name\": \"15bc30b9-fad7-4ef1-9263-ef766496d2c6\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:00:40.843091586Z\",\n      \"endTime\": \"2026-08-07T13:19:40.666609145Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"c4645421d9eeb91087690d2416cb82300c2b51ffeb53e907d01f2076fe2731ef","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"fa7cc051a262a56a5a07d7f638d114a8836b03fd455aef85331260d79e8fd180","test_name":"Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010491e0\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-xtwc9mt2vnz5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5b24d7b-45b9-41ba-8dc7-34abdca66f91\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5b24d7b-45b9-41ba-8dc7-34abdca66f91\",\n      \"name\": \"f5b24d7b-45b9-41ba-8dc7-34abdca66f91\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:09.73986058Z\",\n      \"endTime\": \"2026-08-07T12:59:09.20480396Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010491e0\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-xtwc9mt2vnz5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5b24d7b-45b9-41ba-8dc7-34abdca66f91\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5b24d7b-45b9-41ba-8dc7-34abdca66f91\",\n      \"name\": \"f5b24d7b-45b9-41ba-8dc7-34abdca66f91\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:09.73986058Z\",\n      \"endTime\": \"2026-08-07T12:59:09.20480396Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ca84e39fc0a11a23ff5a16ed73ce768771fa48429077ee4f6f0ad4d86d5e2c75","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"110c23e3f55d6a3fed9ca428846e4eced426c77778556e376226e40255475263","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-nskk9p\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000aa01a0\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-nskk9p: failed waiting for cluster=\"cluster-zstream-4-21-nskk9p\" in resourcegroup=\"rg-zstream-upgrade-4-21-vvm9n5n98j5l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70c3eb67-bd6b-442d-8ffc-c677cd6b4f4e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70c3eb67-bd6b-442d-8ffc-c677cd6b4f4e\",\n      \"name\": \"70c3eb67-bd6b-442d-8ffc-c677cd6b4f4e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:02.222458662Z\",\n      \"endTime\": \"2026-08-07T12:57:09.109215526Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-nskk9p\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000aa01a0\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-nskk9p: failed waiting for cluster=\"cluster-zstream-4-21-nskk9p\" in resourcegroup=\"rg-zstream-upgrade-4-21-vvm9n5n98j5l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70c3eb67-bd6b-442d-8ffc-c677cd6b4f4e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70c3eb67-bd6b-442d-8ffc-c677cd6b4f4e\",\n      \"name\": \"70c3eb67-bd6b-442d-8ffc-c677cd6b4f4e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:02.222458662Z\",\n      \"endTime\": \"2026-08-07T12:57:09.109215526Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"cfd5fc6d6b800f4f3a2ea2f189b52ee942fd6f54689b99625145ee3c61a4da4d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"83a4200526893cc04663aa5c994ace7ad1ba6a311f78a31ff0c17a0d1b38950a","test_name":"Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001082d40\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-rzv65x7zb4gv\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1e2959a-355c-4cc9-8f2a-bc7fdc8c9c47\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1e2959a-355c-4cc9-8f2a-bc7fdc8c9c47\",\n      \"name\": \"e1e2959a-355c-4cc9-8f2a-bc7fdc8c9c47\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:20:04.21517773Z\",\n      \"endTime\": \"2026-08-07T12:39:09.813835734Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001082d40\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-rzv65x7zb4gv\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1e2959a-355c-4cc9-8f2a-bc7fdc8c9c47\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1e2959a-355c-4cc9-8f2a-bc7fdc8c9c47\",\n      \"name\": \"e1e2959a-355c-4cc9-8f2a-bc7fdc8c9c47\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:20:04.21517773Z\",\n      \"endTime\": \"2026-08-07T12:39:09.813835734Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d1ec7b5e2765a91bb90179fe16a369bbdc376ecbeb9d3356a1c70a3c3c6dff44","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"5753c8e666bd0184fc732a31b338fa5c9f46a420adbb9d48190803e8389e9243","test_name":"Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000bd7620\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-487nj864sqw2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ccc22e89-468e-4166-9675-0f30850e86db\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ccc22e89-468e-4166-9675-0f30850e86db\",\n      \"name\": \"ccc22e89-468e-4166-9675-0f30850e86db\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:16:23.87338346Z\",\n      \"endTime\": \"2026-08-07T12:35:29.533043501Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000bd7620\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-487nj864sqw2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ccc22e89-468e-4166-9675-0f30850e86db\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ccc22e89-468e-4166-9675-0f30850e86db\",\n      \"name\": \"ccc22e89-468e-4166-9675-0f30850e86db\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:16:23.87338346Z\",\n      \"endTime\": \"2026-08-07T12:35:29.533043501Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d636c18ac339dd0f729f8e0b54cf59844668155a92f4170a80fa47c2dff11e71","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"f36dbfdb45299387f6797d6dff88f29c060f342480e21c118899e83f61b7662a","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-4t44jpctl6hc/cluster-candidate-4-22-knrlwh should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e444c0\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-knrlwh: failed waiting for cluster=\"cluster-candidate-4-22-knrlwh\" in resourcegroup=\"rg-candidate-4-22-4t44jpctl6hc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0935d9b2-0a34-40a1-881a-e7df70dda369\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0935d9b2-0a34-40a1-881a-e7df70dda369\",\n      \"name\": \"0935d9b2-0a34-40a1-881a-e7df70dda369\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:07.506787964Z\",\n      \"endTime\": \"2026-08-07T12:56:19.249593671Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-4t44jpctl6hc/cluster-candidate-4-22-knrlwh should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e444c0\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-knrlwh: failed waiting for cluster=\"cluster-candidate-4-22-knrlwh\" in resourcegroup=\"rg-candidate-4-22-4t44jpctl6hc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0935d9b2-0a34-40a1-881a-e7df70dda369\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0935d9b2-0a34-40a1-881a-e7df70dda369\",\n      \"name\": \"0935d9b2-0a34-40a1-881a-e7df70dda369\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:37:07.506787964Z\",\n      \"endTime\": \"2026-08-07T12:56:19.249593671Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d92a9c5e8dc38eff0d34269c2dfeb292974f2ca2e89429034fe83b209d42eabb","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"90a6f785605594c10facd804e9b6c549ec615ec97145945fa8ecc7d89f25f1b5","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-smmrpt with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f00760\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-smmrpt: failed waiting for cluster=\"np-version-upgrade-cluster-smmrpt\" in resourcegroup=\"rg-np-version-upgrade-smmrpt-rm2pvmlm7xjd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/747d2108-ebb9-4f04-bf82-b8f5e28f5fb5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/747d2108-ebb9-4f04-bf82-b8f5e28f5fb5\",\n      \"name\": \"747d2108-ebb9-4f04-bf82-b8f5e28f5fb5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:42:31.192514272Z\",\n      \"endTime\": \"2026-08-07T13:01:39.455347528Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-smmrpt with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f00760\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-smmrpt: failed waiting for cluster=\"np-version-upgrade-cluster-smmrpt\" in resourcegroup=\"rg-np-version-upgrade-smmrpt-rm2pvmlm7xjd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/747d2108-ebb9-4f04-bf82-b8f5e28f5fb5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/747d2108-ebb9-4f04-bf82-b8f5e28f5fb5\",\n      \"name\": \"747d2108-ebb9-4f04-bf82-b8f5e28f5fb5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:42:31.192514272Z\",\n      \"endTime\": \"2026-08-07T13:01:39.455347528Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"de03532f037c29850869059cb860d77ee6aaa0140942053ed89228d051c1b929","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"df8f9fc178d598066ef4dc95006c14f9e80e0ca9926db2855cf8740764c4ced5","test_name":"Customer should be able to create an HCP cluster and manage ImageDigestMirrors","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000c02980\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-h9k4gcmkcmwq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6a7d32ec-20e4-409b-8195-73c2167dc2db\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6a7d32ec-20e4-409b-8195-73c2167dc2db\",\n      \"name\": \"6a7d32ec-20e4-409b-8195-73c2167dc2db\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:27.162815114Z\",\n      \"endTime\": \"2026-08-07T12:57:29.025687528Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000c02980\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-h9k4gcmkcmwq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6a7d32ec-20e4-409b-8195-73c2167dc2db\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6a7d32ec-20e4-409b-8195-73c2167dc2db\",\n      \"name\": \"6a7d32ec-20e4-409b-8195-73c2167dc2db\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:27.162815114Z\",\n      \"endTime\": \"2026-08-07T12:57:29.025687528Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"e1dd394ab609fbcbfcd43c4ca8dd09295b5942016202ce9a3bdf04824206ab7a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"0effbd6eaa17b59b272da576065c1b637e7c716a08c26190c0d227da9aa9b61e","test_name":"Customer should be able to lifecycle and confirm external auth on a cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00141e360\u003e: \n    failed to create HCP cluster ea-listtwjc6g: failed waiting for cluster=\"ea-listtwjc6g\" in resourcegroup=\"ea-list-v74txmc8757n\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5910a82b-1d6c-404b-939e-e56f0914be0d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5910a82b-1d6c-404b-939e-e56f0914be0d\",\n      \"name\": \"5910a82b-1d6c-404b-939e-e56f0914be0d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:16.502545192Z\",\n      \"endTime\": \"2026-08-07T12:57:19.129538858Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00141e360\u003e: \n    failed to create HCP cluster ea-listtwjc6g: failed waiting for cluster=\"ea-listtwjc6g\" in resourcegroup=\"ea-list-v74txmc8757n\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5910a82b-1d6c-404b-939e-e56f0914be0d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5910a82b-1d6c-404b-939e-e56f0914be0d\",\n      \"name\": \"5910a82b-1d6c-404b-939e-e56f0914be0d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:16.502545192Z\",\n      \"endTime\": \"2026-08-07T12:57:19.129538858Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"f0482d1005d55c2e6024b7874b31a01dd7b61f4fb30dadbc16d508f803f04510","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"e1f74c7fae16cdd77721dcb44878fa4af55312caf3cac453df7c4fde8bccf7e4","test_name":"Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc000f3d3e0\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d209f7d-0c78-40bb-b797-38e66c520400\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d209f7d-0c78-40bb-b797-38e66c520400\",\n      \"name\": \"4d209f7d-0c78-40bb-b797-38e66c520400\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:15:22.588702197Z\",\n      \"endTime\": \"2026-08-07T12:34:29.768185765Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc000f3d3e0\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d209f7d-0c78-40bb-b797-38e66c520400\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d209f7d-0c78-40bb-b797-38e66c520400\",\n      \"name\": \"4d209f7d-0c78-40bb-b797-38e66c520400\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:15:22.588702197Z\",\n      \"endTime\": \"2026-08-07T12:34:29.768185765Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"f2e9a0be5a0cf3e4b00f210daae2c45f7a4ebb9ebe521263f94d444f3537bfa9","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"8ee1bc48b0cbf269b9f69dca4719f73b8844b81b17eb726bd836ccde827c1788","test_name":"Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-5hvvw9\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e74200\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-5hvvw9: failed waiting for cluster=\"cp-ystream-upgrade-4-21-5hvvw9\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-pjr68ww6vmvz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0171d88e-a7a4-462d-bc1b-37b85dcd4e8e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0171d88e-a7a4-462d-bc1b-37b85dcd4e8e\",\n      \"name\": \"0171d88e-a7a4-462d-bc1b-37b85dcd4e8e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:02.288277557Z\",\n      \"endTime\": \"2026-08-07T12:57:09.088329463Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-5hvvw9\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e74200\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-5hvvw9: failed waiting for cluster=\"cp-ystream-upgrade-4-21-5hvvw9\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-pjr68ww6vmvz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0171d88e-a7a4-462d-bc1b-37b85dcd4e8e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0171d88e-a7a4-462d-bc1b-37b85dcd4e8e\",\n      \"name\": \"0171d88e-a7a4-462d-bc1b-37b85dcd4e8e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:38:02.288277557Z\",\n      \"endTime\": \"2026-08-07T12:57:09.088329463Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"f5f872784ab21ab7950025e4d3602d88814edc582d08b6964cece685b3f8a5bb","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"915bd047c73083606b52b23bc886dfa52d2689e3f5d40ddf0155e88d81861ff8","test_name":"Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0007f0560\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rpgpgbkmgvgz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1c3c751b-ca8f-4403-a5f8-b374158b8b78\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1c3c751b-ca8f-4403-a5f8-b374158b8b78\",\n      \"name\": \"1c3c751b-ca8f-4403-a5f8-b374158b8b78\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:18.063563284Z\",\n      \"endTime\": \"2026-08-07T12:59:19.166687382Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0007f0560\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-rpgpgbkmgvgz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1c3c751b-ca8f-4403-a5f8-b374158b8b78\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1c3c751b-ca8f-4403-a5f8-b374158b8b78\",\n      \"name\": \"1c3c751b-ca8f-4403-a5f8-b374158b8b78\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T12:40:18.063563284Z\",\n      \"endTime\": \"2026-08-07T12:59:19.166687382Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ffdea816f383c1c977add8cd100304b3a4ddddc58aa5fda18811373f5ed1b11c","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085696426184019968","occurred_at":"2026-08-07T11:55:52Z","failed_at":"e2e","signature_id":"f48226e13a0777afda32e391bdf0b9d582005bd26b420245a79d46f56af02702","test_name":"Customer should not be able to perform various invalid operations on cluster resources","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001144b00\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-nst7m5qfdprc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2b8b9a1-f066-4b6d-b2fc-9afc147fd254\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2b8b9a1-f066-4b6d-b2fc-9afc147fd254\",\n      \"name\": \"b2b8b9a1-f066-4b6d-b2fc-9afc147fd254\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:01:16.112788449Z\",\n      \"endTime\": \"2026-08-07T13:20:20.621319184Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001144b00\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-nst7m5qfdprc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2b8b9a1-f066-4b6d-b2fc-9afc147fd254\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2b8b9a1-f066-4b6d-b2fc-9afc147fd254\",\n      \"name\": \"b2b8b9a1-f066-4b6d-b2fc-9afc147fd254\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T13:01:16.112788449Z\",\n      \"endTime\": \"2026-08-07T13:20:20.621319184Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}}],"failure_pattern_summary":{"signature_count":44,"failure_pattern_ids":["0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b"],"matched_occurrences":44,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T09:46:32Z"},"failed_at":["e2e"],"failed_test_count":44,"occurrences":[{"row_id":"031824c7663b4830f3808662affc202d0a452d2402e894dd7cce939bca2978bb","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"f921680850eb9fbb43a1765affc2dc06426bfd5755327ff086c6893fc938a211","test_name":"Customer should be able to create a cluster with an external auth config and get the external auth config","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e82ac0\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch8b7jklbngh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c04792d7-bdd8-4393-938b-01f39195193e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c04792d7-bdd8-4393-938b-01f39195193e\",\n      \"name\": \"c04792d7-bdd8-4393-938b-01f39195193e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:35.331416497Z\",\n      \"endTime\": \"2026-08-07T10:30:54.610840546Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e82ac0\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-ch8b7jklbngh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c04792d7-bdd8-4393-938b-01f39195193e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c04792d7-bdd8-4393-938b-01f39195193e\",\n      \"name\": \"c04792d7-bdd8-4393-938b-01f39195193e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:35.331416497Z\",\n      \"endTime\": \"2026-08-07T10:30:54.610840546Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"06e038a6565587f6e16609790510122fbd08a8ea8f41719fdd6dbf43a5558267","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"19bfb5c7ff0414868479f61ffa6d9a01c26c4f15b5099a196c64ef2630ce058a","test_name":"Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a926e0\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-szp4g65687p5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/53a5aa62-24d7-4be4-b8ae-5248834cdbd4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/53a5aa62-24d7-4be4-b8ae-5248834cdbd4\",\n      \"name\": \"53a5aa62-24d7-4be4-b8ae-5248834cdbd4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:54.719443831Z\",\n      \"endTime\": \"2026-08-07T10:32:54.896558415Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a926e0\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-szp4g65687p5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/53a5aa62-24d7-4be4-b8ae-5248834cdbd4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/53a5aa62-24d7-4be4-b8ae-5248834cdbd4\",\n      \"name\": \"53a5aa62-24d7-4be4-b8ae-5248834cdbd4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:54.719443831Z\",\n      \"endTime\": \"2026-08-07T10:32:54.896558415Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0ecef170fe56a628e592166050696e4c9164d33caa740ee376d4d15d0aba8a86","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"9a8377eda69c2082b7682e3e21f2ceccb5b8843896e7b97f4d011887b4e426db","test_name":"Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006d87e0\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rg4dnbmx8645\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c4241b6-4c51-4e09-b24a-10ea330933bc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c4241b6-4c51-4e09-b24a-10ea330933bc\",\n      \"name\": \"9c4241b6-4c51-4e09-b24a-10ea330933bc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:51:21.277228592Z\",\n      \"endTime\": \"2026-08-07T10:10:22.838228964Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006d87e0\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rg4dnbmx8645\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c4241b6-4c51-4e09-b24a-10ea330933bc\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9c4241b6-4c51-4e09-b24a-10ea330933bc\",\n      \"name\": \"9c4241b6-4c51-4e09-b24a-10ea330933bc\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:51:21.277228592Z\",\n      \"endTime\": \"2026-08-07T10:10:22.838228964Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"19608031e1559f2f40883cab676d291cc5f94460114d1e1b504b2fff6b6c4110","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"5745a2c879e8e7083697b858b9702751549e60fe482c52bd687da97963f1520a","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-rfxpbl\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e90500\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-rfxpbl: failed waiting for cluster=\"cluster-zstream-4-22-rfxpbl\" in resourcegroup=\"rg-zstream-upgrade-4-22-jw5ckb7cmw7m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1bc6d749-0875-40a4-b76a-fa74830cfb65\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1bc6d749-0875-40a4-b76a-fa74830cfb65\",\n      \"name\": \"1bc6d749-0875-40a4-b76a-fa74830cfb65\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:21.722930955Z\",\n      \"endTime\": \"2026-08-07T10:30:34.745049308Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-rfxpbl\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e90500\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-rfxpbl: failed waiting for cluster=\"cluster-zstream-4-22-rfxpbl\" in resourcegroup=\"rg-zstream-upgrade-4-22-jw5ckb7cmw7m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1bc6d749-0875-40a4-b76a-fa74830cfb65\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1bc6d749-0875-40a4-b76a-fa74830cfb65\",\n      \"name\": \"1bc6d749-0875-40a4-b76a-fa74830cfb65\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:21.722930955Z\",\n      \"endTime\": \"2026-08-07T10:30:34.745049308Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"1e77b1d27687d9a9695248477ba5eab7562ff00fc7f21fd4aa0b7f24cf80b81a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"2e845ba485ca50a17fa49923c46eb7af267d08b92265116ef2d24ce2ee2b73a6","test_name":"Customer should be able to create an HCP cluster and manage pull secrets","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00118cb20\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-js6fd6j22j94\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34937a35-7ebb-468d-98d0-20009052cbec\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34937a35-7ebb-468d-98d0-20009052cbec\",\n      \"name\": \"34937a35-7ebb-468d-98d0-20009052cbec\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:39.737544751Z\",\n      \"endTime\": \"2026-08-07T10:07:52.742732935Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00118cb20\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-js6fd6j22j94\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34937a35-7ebb-468d-98d0-20009052cbec\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34937a35-7ebb-468d-98d0-20009052cbec\",\n      \"name\": \"34937a35-7ebb-468d-98d0-20009052cbec\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:39.737544751Z\",\n      \"endTime\": \"2026-08-07T10:07:52.742732935Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"245a2686bb9415f365bf5c85b91cf882423bbdeb57d482d4de2af29275ded186","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"f185b53f7df26352cf08dbaf3cd46c6af0307d7457a94080bdb2b6a780478c48","test_name":"Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-z6bqkx with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009eee40\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-z6bqkx: failed waiting for cluster=\"np-version-upgrade-cluster-z6bqkx\" in resourcegroup=\"rg-np-version-upgrade-z6bqkx-b9qlfsckw56h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97bef450-5aa6-41d6-aefb-6afe651e189d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97bef450-5aa6-41d6-aefb-6afe651e189d\",\n      \"name\": \"97bef450-5aa6-41d6-aefb-6afe651e189d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:19.571495404Z\",\n      \"endTime\": \"2026-08-07T10:52:23.715250036Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-z6bqkx with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009eee40\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-z6bqkx: failed waiting for cluster=\"np-version-upgrade-cluster-z6bqkx\" in resourcegroup=\"rg-np-version-upgrade-z6bqkx-b9qlfsckw56h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97bef450-5aa6-41d6-aefb-6afe651e189d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97bef450-5aa6-41d6-aefb-6afe651e189d\",\n      \"name\": \"97bef450-5aa6-41d6-aefb-6afe651e189d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:19.571495404Z\",\n      \"endTime\": \"2026-08-07T10:52:23.715250036Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"2c06e93200f4f5c40d05d4c1cde171835d32f5a90436ef2b4c9b0be8818c9b56","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"3967c2d236919561e32c384a3902a52a7f471e7c81188ccf196ca8ad9f3277e2","test_name":"Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-2297ld\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00101eae0\u003e: \n    failed to create HCP cluster np-noedge-2297ld: failed waiting for cluster=\"np-noedge-2297ld\" in resourcegroup=\"rg-np-noedge-2297ld-5lrft7w8zkkw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3664ba8-ebcd-4d80-8ee2-85ff57a973ed\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3664ba8-ebcd-4d80-8ee2-85ff57a973ed\",\n      \"name\": \"c3664ba8-ebcd-4d80-8ee2-85ff57a973ed\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:44.071863603Z\",\n      \"endTime\": \"2026-08-07T10:52:53.661061466Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-2297ld\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00101eae0\u003e: \n    failed to create HCP cluster np-noedge-2297ld: failed waiting for cluster=\"np-noedge-2297ld\" in resourcegroup=\"rg-np-noedge-2297ld-5lrft7w8zkkw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3664ba8-ebcd-4d80-8ee2-85ff57a973ed\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c3664ba8-ebcd-4d80-8ee2-85ff57a973ed\",\n      \"name\": \"c3664ba8-ebcd-4d80-8ee2-85ff57a973ed\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:44.071863603Z\",\n      \"endTime\": \"2026-08-07T10:52:53.661061466Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"31171024c1d85209e9926463ea87402257e6e0e9a83067bababa044c95b1d759","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"5372ebf626ff54f6a8ae66739f901a51e06b3b89ebc56584554e118159e1957d","test_name":"Customer should not be able to reuse subnets and NSGs between clusters","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001510a60\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-rlwrf5psdwbw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2f39581-1a08-49d0-9aa4-ccceaa0c2827\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2f39581-1a08-49d0-9aa4-ccceaa0c2827\",\n      \"name\": \"c2f39581-1a08-49d0-9aa4-ccceaa0c2827\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:52.156305081Z\",\n      \"endTime\": \"2026-08-07T10:07:52.85878894Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001510a60\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-rlwrf5psdwbw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2f39581-1a08-49d0-9aa4-ccceaa0c2827\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2f39581-1a08-49d0-9aa4-ccceaa0c2827\",\n      \"name\": \"c2f39581-1a08-49d0-9aa4-ccceaa0c2827\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:52.156305081Z\",\n      \"endTime\": \"2026-08-07T10:07:52.85878894Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"3d1551f4eda822304cd50ee0639021002fdad2811723dbcadbe0e493c6b6226c","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"727abaa3777f6a2437bb9c5a6df347940e4ae45bcba8d1b12d46ed6cbf87ff62","test_name":"Customer should be able to create node pools with ARM64-based VMs","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000881140\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-vmnjrlk2j4jd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/60447c67-ab89-4b49-8ca4-6c1023facdac\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/60447c67-ab89-4b49-8ca4-6c1023facdac\",\n      \"name\": \"60447c67-ab89-4b49-8ca4-6c1023facdac\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:39.225641432Z\",\n      \"endTime\": \"2026-08-07T10:07:52.712144184Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000881140\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-vmnjrlk2j4jd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/60447c67-ab89-4b49-8ca4-6c1023facdac\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/60447c67-ab89-4b49-8ca4-6c1023facdac\",\n      \"name\": \"60447c67-ab89-4b49-8ca4-6c1023facdac\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:39.225641432Z\",\n      \"endTime\": \"2026-08-07T10:07:52.712144184Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4202bf713c63ef46b6fed94377efd7c65b46903c248947132c54c69c5ed5ea9b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"f45559561605752a0951cc8fe81061144b8b0db680c0315c8529be56384dd63b","test_name":"Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010d0a60\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-mdnjvhdws59j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3bf0c10-5629-4f24-a157-2205cdbf4105\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3bf0c10-5629-4f24-a157-2205cdbf4105\",\n      \"name\": \"f3bf0c10-5629-4f24-a157-2205cdbf4105\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:50:22.24706297Z\",\n      \"endTime\": \"2026-08-07T10:09:22.86494678Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010d0a60\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-mdnjvhdws59j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3bf0c10-5629-4f24-a157-2205cdbf4105\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f3bf0c10-5629-4f24-a157-2205cdbf4105\",\n      \"name\": \"f3bf0c10-5629-4f24-a157-2205cdbf4105\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:50:22.24706297Z\",\n      \"endTime\": \"2026-08-07T10:09:22.86494678Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4ac216217ead2790375c767c1b4e065f664862ae71fd86d429342d9c717c697e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"3a0150a6885c417dda7f4decf203b51070a91036a831fc355e7514e46c9e309e","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-56gzbn2tpp9t/cluster-candidate-4-21-pzpt9b should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004b1800\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-pzpt9b: failed waiting for cluster=\"cluster-candidate-4-21-pzpt9b\" in resourcegroup=\"rg-candidate-4-21-56gzbn2tpp9t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/be7b8d5c-1c0d-4175-84ef-46e3ec0dff44\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/be7b8d5c-1c0d-4175-84ef-46e3ec0dff44\",\n      \"name\": \"be7b8d5c-1c0d-4175-84ef-46e3ec0dff44\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:10:50.612137852Z\",\n      \"endTime\": \"2026-08-07T10:29:54.453114842Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-56gzbn2tpp9t/cluster-candidate-4-21-pzpt9b should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004b1800\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-pzpt9b: failed waiting for cluster=\"cluster-candidate-4-21-pzpt9b\" in resourcegroup=\"rg-candidate-4-21-56gzbn2tpp9t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/be7b8d5c-1c0d-4175-84ef-46e3ec0dff44\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/be7b8d5c-1c0d-4175-84ef-46e3ec0dff44\",\n      \"name\": \"be7b8d5c-1c0d-4175-84ef-46e3ec0dff44\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:10:50.612137852Z\",\n      \"endTime\": \"2026-08-07T10:29:54.453114842Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4c236deca276218b41b9544b4878318f5407939132e4f030860abe6d81f0e90e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"51312044ea6ef71b5dd8622792e199579fa4beb715251c09dfa52d4ddb709ff4","test_name":"Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc00168f500\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc71b9f2-bf54-41bb-86e5-27a6fba9ae46\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc71b9f2-bf54-41bb-86e5-27a6fba9ae46\",\n      \"name\": \"dc71b9f2-bf54-41bb-86e5-27a6fba9ae46\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:50:17.184853919Z\",\n      \"endTime\": \"2026-08-07T10:09:22.870507096Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc00168f500\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc71b9f2-bf54-41bb-86e5-27a6fba9ae46\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc71b9f2-bf54-41bb-86e5-27a6fba9ae46\",\n      \"name\": \"dc71b9f2-bf54-41bb-86e5-27a6fba9ae46\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:50:17.184853919Z\",\n      \"endTime\": \"2026-08-07T10:09:22.870507096Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"55604ece2b488e6b8f5a8c34e0418455c917ed6d8973cbb2a1353aa02a5f712a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"d5b61440e1de74f581d4662ffbac765fd2d3ad1d4b3c25ac4352028e8197e7e6","test_name":"Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-gpskql\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b49200\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-gpskql: failed waiting for cluster=\"cp-ystream-upgrade-4-21-gpskql\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-87jrx9rdm6qc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626a5876-ffb0-43d1-8486-1de57be72ac2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626a5876-ffb0-43d1-8486-1de57be72ac2\",\n      \"name\": \"626a5876-ffb0-43d1-8486-1de57be72ac2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:31.597655131Z\",\n      \"endTime\": \"2026-08-07T10:30:34.774783283Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-gpskql\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b49200\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-gpskql: failed waiting for cluster=\"cp-ystream-upgrade-4-21-gpskql\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-87jrx9rdm6qc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626a5876-ffb0-43d1-8486-1de57be72ac2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626a5876-ffb0-43d1-8486-1de57be72ac2\",\n      \"name\": \"626a5876-ffb0-43d1-8486-1de57be72ac2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:31.597655131Z\",\n      \"endTime\": \"2026-08-07T10:30:34.774783283Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"5ed36a773380d8926c841a605b7a0bb719c48d01c6fb4822be4d292453d93367","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"2d1a08d4693343c72e5091b5791d71fba5dcfaa38a5297a85086aed999fc7c0b","test_name":"HCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001cbc8e0\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-b4ltf68p8wsb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a005dc4b-3cd7-4890-a4f8-92f51126c76a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a005dc4b-3cd7-4890-a4f8-92f51126c76a\",\n      \"name\": \"a005dc4b-3cd7-4890-a4f8-92f51126c76a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:25.93255187Z\",\n      \"endTime\": \"2026-08-07T10:32:34.943121697Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001cbc8e0\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-b4ltf68p8wsb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a005dc4b-3cd7-4890-a4f8-92f51126c76a\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a005dc4b-3cd7-4890-a4f8-92f51126c76a\",\n      \"name\": \"a005dc4b-3cd7-4890-a4f8-92f51126c76a\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:25.93255187Z\",\n      \"endTime\": \"2026-08-07T10:32:34.943121697Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"5f50448a27570d08c5c9fa9d33c58fbc676fd1fe675112ef5ebc671575aab641","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"e637df44fb96755b499f362cd8119a8c45b352805fb7779d4f3a76b1f00e2ebf","test_name":"Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-n7mmck at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009e7140\u003e: \n    failed to create HCP cluster np-skip-minor-n7mmck: failed waiting for cluster=\"np-skip-minor-n7mmck\" in resourcegroup=\"rg-np-skip-minor-n7mmck-jd4n984jp22b\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/18ee5978-f00a-4f14-b403-4250519120aa\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/18ee5978-f00a-4f14-b403-4250519120aa\",\n      \"name\": \"18ee5978-f00a-4f14-b403-4250519120aa\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:49.736150552Z\",\n      \"endTime\": \"2026-08-07T10:52:53.673838751Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-n7mmck at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009e7140\u003e: \n    failed to create HCP cluster np-skip-minor-n7mmck: failed waiting for cluster=\"np-skip-minor-n7mmck\" in resourcegroup=\"rg-np-skip-minor-n7mmck-jd4n984jp22b\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/18ee5978-f00a-4f14-b403-4250519120aa\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/18ee5978-f00a-4f14-b403-4250519120aa\",\n      \"name\": \"18ee5978-f00a-4f14-b403-4250519120aa\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:49.736150552Z\",\n      \"endTime\": \"2026-08-07T10:52:53.673838751Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"606d05228ae7c14ded613966d5cae49c8021662e24d685d5625e78be96826a78","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"0584690cc960d6cb7ebbc39b87c4a5b80a52e15ac8d7a3f35baac916cdf3559d","test_name":"Authorized CIDRs Connectivity should allow API access only from authorized VM IP","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001200860\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-vn2996fxlbpm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b75e483-de81-4efc-82c5-b47026a3b704\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b75e483-de81-4efc-82c5-b47026a3b704\",\n      \"name\": \"4b75e483-de81-4efc-82c5-b47026a3b704\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:49:42.488006758Z\",\n      \"endTime\": \"2026-08-07T10:08:42.868675079Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001200860\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-vn2996fxlbpm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b75e483-de81-4efc-82c5-b47026a3b704\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b75e483-de81-4efc-82c5-b47026a3b704\",\n      \"name\": \"4b75e483-de81-4efc-82c5-b47026a3b704\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:49:42.488006758Z\",\n      \"endTime\": \"2026-08-07T10:08:42.868675079Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"62546003bafd51b1d913ef6ba14e2094efe0dbe515986e4e9812036125c33f7e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"517b14c8289f1bebb11b4e481782f400cc44a0cda73bef41a3582eca65282b73","test_name":"Customer should respect cluster-wide node limits with nodepool autoscaling","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e38640\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-mzrf27h6l5ng\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/001aeb44-144c-4115-8e72-668532f4c086\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/001aeb44-144c-4115-8e72-668532f4c086\",\n      \"name\": \"001aeb44-144c-4115-8e72-668532f4c086\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:59.985884034Z\",\n      \"endTime\": \"2026-08-07T10:33:04.92002595Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e38640\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-mzrf27h6l5ng\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/001aeb44-144c-4115-8e72-668532f4c086\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/001aeb44-144c-4115-8e72-668532f4c086\",\n      \"name\": \"001aeb44-144c-4115-8e72-668532f4c086\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:59.985884034Z\",\n      \"endTime\": \"2026-08-07T10:33:04.92002595Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"632b06cfe3018335214ea0865431348c23d3fd66bb605433f5ebf5405c1fec6e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"475729c5eff4fdf1069558b0313a4357c72f51e83a8ef3eece60044f24ae425b","test_name":"Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000fb4f80\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-xl859w49w6td\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37dc07e3-f9d3-4296-88e9-35c48d7c6e12\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37dc07e3-f9d3-4296-88e9-35c48d7c6e12\",\n      \"name\": \"37dc07e3-f9d3-4296-88e9-35c48d7c6e12\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:14:01.706234743Z\",\n      \"endTime\": \"2026-08-07T10:33:14.841620435Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000fb4f80\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-xl859w49w6td\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37dc07e3-f9d3-4296-88e9-35c48d7c6e12\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/37dc07e3-f9d3-4296-88e9-35c48d7c6e12\",\n      \"name\": \"37dc07e3-f9d3-4296-88e9-35c48d7c6e12\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:14:01.706234743Z\",\n      \"endTime\": \"2026-08-07T10:33:14.841620435Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"6341ea823f7fe05787db07f2902ca5aaa7255659802489e5676109fe3b5ffe4b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"c22b9fb8e8d4856a0db361c1177e97c13205cfd46b715e44573b9b8063176606","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-vc9tf6 with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001270360\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-vc9tf6: failed waiting for cluster=\"np-version-upgrade-cluster-vc9tf6\" in resourcegroup=\"rg-np-version-upgrade-vc9tf6-f2jwtg9fcrlh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c8e02d70-7d8b-45b8-aa19-bf25830f66a8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c8e02d70-7d8b-45b8-aa19-bf25830f66a8\",\n      \"name\": \"c8e02d70-7d8b-45b8-aa19-bf25830f66a8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:45.568930171Z\",\n      \"endTime\": \"2026-08-07T10:52:53.637127738Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-vc9tf6 with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001270360\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-vc9tf6: failed waiting for cluster=\"np-version-upgrade-cluster-vc9tf6\" in resourcegroup=\"rg-np-version-upgrade-vc9tf6-f2jwtg9fcrlh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c8e02d70-7d8b-45b8-aa19-bf25830f66a8\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c8e02d70-7d8b-45b8-aa19-bf25830f66a8\",\n      \"name\": \"c8e02d70-7d8b-45b8-aa19-bf25830f66a8\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:45.568930171Z\",\n      \"endTime\": \"2026-08-07T10:52:53.637127738Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"699ba8ed2a18412f20e718aec0ab977cf39e95bd5deeb7224155eb0a6411dd09","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"d3a04983968add43cc36112e51d894431e92466e7c0540a0919628fc95b70c5c","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-4hw9h4nwskb7/cluster-candidate-4-20-cvl8rn should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0012c04a0\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-cvl8rn: failed waiting for cluster=\"cluster-candidate-4-20-cvl8rn\" in resourcegroup=\"rg-candidate-4-20-4hw9h4nwskb7\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/827108fe-13cd-4a3b-b26d-23346ce652c7\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/827108fe-13cd-4a3b-b26d-23346ce652c7\",\n      \"name\": \"827108fe-13cd-4a3b-b26d-23346ce652c7\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:10:32.475407407Z\",\n      \"endTime\": \"2026-08-07T10:29:44.685540326Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-4hw9h4nwskb7/cluster-candidate-4-20-cvl8rn should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0012c04a0\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-cvl8rn: failed waiting for cluster=\"cluster-candidate-4-20-cvl8rn\" in resourcegroup=\"rg-candidate-4-20-4hw9h4nwskb7\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/827108fe-13cd-4a3b-b26d-23346ce652c7\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/827108fe-13cd-4a3b-b26d-23346ce652c7\",\n      \"name\": \"827108fe-13cd-4a3b-b26d-23346ce652c7\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:10:32.475407407Z\",\n      \"endTime\": \"2026-08-07T10:29:44.685540326Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"6a91aea2e99d648d267ac0715be991b2b81eecb2d17c27cdf812a08cd507936e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"86c5a9d759d82e92db663ef6e7d44c8f4e8362201b64bd75057deaeb2bbbf076","test_name":"Customer should create an HCP cluster and validate TLS certificates","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ed52a0\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-pr7xwqsb7nzc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50e5d5d6-a1f7-4a4e-9567-b06d0f1d3015\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50e5d5d6-a1f7-4a4e-9567-b06d0f1d3015\",\n      \"name\": \"50e5d5d6-a1f7-4a4e-9567-b06d0f1d3015\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:39.975146966Z\",\n      \"endTime\": \"2026-08-07T10:07:52.711104202Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ed52a0\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-pr7xwqsb7nzc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50e5d5d6-a1f7-4a4e-9567-b06d0f1d3015\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/50e5d5d6-a1f7-4a4e-9567-b06d0f1d3015\",\n      \"name\": \"50e5d5d6-a1f7-4a4e-9567-b06d0f1d3015\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:39.975146966Z\",\n      \"endTime\": \"2026-08-07T10:07:52.711104202Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"6c0c8d5a2d970054ebbc0856bb4dac9a43c3983c63771b81effdd30d308493c8","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"0a44f8a7a3cbc3fdd31ae266795e3abb2acbcc1e6c5d1351f3e9df9f2be3e52d","test_name":"Customer should be able to create an HCP cluster then delete it by deleting the customer resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000606460\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-r84m756rpdj6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86c98006-48d1-4e12-b5d4-fb197a06add9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86c98006-48d1-4e12-b5d4-fb197a06add9\",\n      \"name\": \"86c98006-48d1-4e12-b5d4-fb197a06add9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:50.342030553Z\",\n      \"endTime\": \"2026-08-07T10:07:52.753488177Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000606460\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-r84m756rpdj6\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86c98006-48d1-4e12-b5d4-fb197a06add9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/86c98006-48d1-4e12-b5d4-fb197a06add9\",\n      \"name\": \"86c98006-48d1-4e12-b5d4-fb197a06add9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:50.342030553Z\",\n      \"endTime\": \"2026-08-07T10:07:52.753488177Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"6f526af0bae215b98b66f99532031d73db608d4a6fa389160589ffd1f3d676ee","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"b105c7c47ba5e712674a6ff85a455da04bf17d6e5e30c473dc16a798e04863af","test_name":"Customer should be able to create an HCP cluster with back-level version 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:150]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0018aa840\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-p56vtnn4kcqf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-p56vtnn4kcqf, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:150]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc0018aa840\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-p56vtnn4kcqf\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-p56vtnn4kcqf, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-br948jjgf9df\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-br948jjgf9df, error: context deadline exceeded"}},{"row_id":"6f608988ce33eb85dd937a891dcd6205f673529985173f87190e2ed42c20ab91","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"f30d51747cb4b6463ed1d0450832421aeb2f9a6d4829e37dcd4549d4a700c194","test_name":"Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d61d40\u003e: \n    failed to create HCP cluster np-dg-minor-87tw69: failed waiting for cluster=\"np-dg-minor-87tw69\" in resourcegroup=\"rg-np-dg-minor-87tw69-tlhrj8dk6sch\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8bf9a329-fe63-4a3c-98d8-25b4ad89a6c4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8bf9a329-fe63-4a3c-98d8-25b4ad89a6c4\",\n      \"name\": \"8bf9a329-fe63-4a3c-98d8-25b4ad89a6c4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:34:18.807431056Z\",\n      \"endTime\": \"2026-08-07T10:53:23.658461744Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d61d40\u003e: \n    failed to create HCP cluster np-dg-minor-87tw69: failed waiting for cluster=\"np-dg-minor-87tw69\" in resourcegroup=\"rg-np-dg-minor-87tw69-tlhrj8dk6sch\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8bf9a329-fe63-4a3c-98d8-25b4ad89a6c4\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8bf9a329-fe63-4a3c-98d8-25b4ad89a6c4\",\n      \"name\": \"8bf9a329-fe63-4a3c-98d8-25b4ad89a6c4\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:34:18.807431056Z\",\n      \"endTime\": \"2026-08-07T10:53:23.658461744Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"711af2f723bacb94656289a60757fe83168c819069e852843cbb343dc6d8eed0","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"14ccaa384f7aeb9c0245951bd88cf53e581b39b24ee9a8016c75c332cf1c49bb","test_name":"Customer should be able to create an HCP cluster and manage ImageDigestMirrors","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f77520\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-kjdl8bbbghqp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d17cf2a5-a1d9-4df8-92d0-f5f450064ac9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d17cf2a5-a1d9-4df8-92d0-f5f450064ac9\",\n      \"name\": \"d17cf2a5-a1d9-4df8-92d0-f5f450064ac9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:25.274529006Z\",\n      \"endTime\": \"2026-08-07T10:32:34.882192239Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f77520\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-kjdl8bbbghqp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d17cf2a5-a1d9-4df8-92d0-f5f450064ac9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d17cf2a5-a1d9-4df8-92d0-f5f450064ac9\",\n      \"name\": \"d17cf2a5-a1d9-4df8-92d0-f5f450064ac9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:25.274529006Z\",\n      \"endTime\": \"2026-08-07T10:32:34.882192239Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"788227ed6e7fe79199498e3f7a19f23c067a3b6a4fc24cfcd1e94d9027564f43","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"cddd87c73995ac00d14eec612938a0ee88da58e32758d73fb32ed8a3438b3927","test_name":"Customer should be able to create an HCP cluster with Image Registry not present","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001004960\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-m7x5ctwhzrm9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a70be4fc-ead7-49c5-8400-6c5ae4505f03\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a70be4fc-ead7-49c5-8400-6c5ae4505f03\",\n      \"name\": \"a70be4fc-ead7-49c5-8400-6c5ae4505f03\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:45.552336892Z\",\n      \"endTime\": \"2026-08-07T10:32:54.853772784Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001004960\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-m7x5ctwhzrm9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a70be4fc-ead7-49c5-8400-6c5ae4505f03\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a70be4fc-ead7-49c5-8400-6c5ae4505f03\",\n      \"name\": \"a70be4fc-ead7-49c5-8400-6c5ae4505f03\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:45.552336892Z\",\n      \"endTime\": \"2026-08-07T10:32:54.853772784Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"82f63a1ff5ac550163ea267c70e0c070071f0a8fb8ca7ee6df058215513bd0dc","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"9bde0735ae665dc5f3d378c45583f1fa1985475792ff363db63f4d3a8010321e","test_name":"Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00114c9a0\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-hlk7qpvd8s5l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1a322ece-933b-4098-b6eb-bee848107b36\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1a322ece-933b-4098-b6eb-bee848107b36\",\n      \"name\": \"1a322ece-933b-4098-b6eb-bee848107b36\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:52:19.30313473Z\",\n      \"endTime\": \"2026-08-07T10:11:22.864006097Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00114c9a0\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-hlk7qpvd8s5l\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1a322ece-933b-4098-b6eb-bee848107b36\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1a322ece-933b-4098-b6eb-bee848107b36\",\n      \"name\": \"1a322ece-933b-4098-b6eb-bee848107b36\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:52:19.30313473Z\",\n      \"endTime\": \"2026-08-07T10:11:22.864006097Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"878f9994830661a2663a243577474d2dc8c5a8a17462e6a0ad52e042cacd131c","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"263177b6d5c0fa8191c9405b3b366a4bc89e2a379a984ade9da0d02532d9bd22","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-rq5dvn\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010101a0\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-rq5dvn: failed waiting for cluster=\"cluster-zstream-4-20-rq5dvn\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrw2jsrdmgkg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/20fcafdb-670d-4f1f-9a29-13d525cac1c0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/20fcafdb-670d-4f1f-9a29-13d525cac1c0\",\n      \"name\": \"20fcafdb-670d-4f1f-9a29-13d525cac1c0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:14.811209323Z\",\n      \"endTime\": \"2026-08-07T10:30:14.422653776Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-rq5dvn\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010101a0\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-rq5dvn: failed waiting for cluster=\"cluster-zstream-4-20-rq5dvn\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrw2jsrdmgkg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/20fcafdb-670d-4f1f-9a29-13d525cac1c0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/20fcafdb-670d-4f1f-9a29-13d525cac1c0\",\n      \"name\": \"20fcafdb-670d-4f1f-9a29-13d525cac1c0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:14.811209323Z\",\n      \"endTime\": \"2026-08-07T10:30:14.422653776Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"8a17cfab8cb59d5299e11137ef079821c479c9b1d9f0ae9131c115428d682fa8","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"d712d5ce17264f0ef550496665e2a7bd7d67dfac479ff00deed5bcb5e610f7c9","test_name":"ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.003s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.003s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","failure_pattern":"cluster entered terminal Failed state after role assignment deployment","search_query":"cluster entered terminal Failed state after role assignment deployment"}},{"row_id":"96ba2fb094d1ce3c3cfc0c6da1186efb3d9e1475e9129ff3db6a3fdb976f5297","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"14c80964b9874c9081110be783c8e717ab9db60e5a66ec45327b015be13e84aa","test_name":"Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed\nfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed","failure_pattern_match":{"status":"clustered","failure_pattern_id":"e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b","failure_pattern":"Cluster provisioning failed","search_query":"Cluster provisioning failed"}},{"row_id":"9d240c5549790dd196c830d04f60d5a54f80fb4efda130478b91ec59e240fa8b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"e86ec2d76390b9792f709a881237048c06912b1eadb050ede53acd46bbf17e68","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-v26ggln4q2m5/cluster-candidate-4-22-kb8g7x should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000922bc0\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-kb8g7x: failed waiting for cluster=\"cluster-candidate-4-22-kb8g7x\" in resourcegroup=\"rg-candidate-4-22-v26ggln4q2m5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fb344d3-636a-4b8b-8ae6-a188bc815e76\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fb344d3-636a-4b8b-8ae6-a188bc815e76\",\n      \"name\": \"0fb344d3-636a-4b8b-8ae6-a188bc815e76\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:25.406278662Z\",\n      \"endTime\": \"2026-08-07T10:30:34.70125686Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-v26ggln4q2m5/cluster-candidate-4-22-kb8g7x should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000922bc0\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-kb8g7x: failed waiting for cluster=\"cluster-candidate-4-22-kb8g7x\" in resourcegroup=\"rg-candidate-4-22-v26ggln4q2m5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fb344d3-636a-4b8b-8ae6-a188bc815e76\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fb344d3-636a-4b8b-8ae6-a188bc815e76\",\n      \"name\": \"0fb344d3-636a-4b8b-8ae6-a188bc815e76\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:25.406278662Z\",\n      \"endTime\": \"2026-08-07T10:30:34.70125686Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"b468a25ad9f64250c5e2d1e22a8e104d2d6ca329c3eb522125baccaf7b20102b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"2f65f944e30f11b17ca6554e0e42e566605d67602b2e3035b797549bf8fa6a91","test_name":"Customer should be able to lifecycle and confirm external auth on a cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001095180\u003e: \n    failed to create HCP cluster ea-listcz7v7j: failed waiting for cluster=\"ea-listcz7v7j\" in resourcegroup=\"ea-list-4lk5hvh7j8m9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/008fa357-5285-483c-813e-4b31d048ccce\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/008fa357-5285-483c-813e-4b31d048ccce\",\n      \"name\": \"008fa357-5285-483c-813e-4b31d048ccce\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:24.992799674Z\",\n      \"endTime\": \"2026-08-07T10:32:24.814909242Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001095180\u003e: \n    failed to create HCP cluster ea-listcz7v7j: failed waiting for cluster=\"ea-listcz7v7j\" in resourcegroup=\"ea-list-4lk5hvh7j8m9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/008fa357-5285-483c-813e-4b31d048ccce\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/008fa357-5285-483c-813e-4b31d048ccce\",\n      \"name\": \"008fa357-5285-483c-813e-4b31d048ccce\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:13:24.992799674Z\",\n      \"endTime\": \"2026-08-07T10:32:24.814909242Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"bf7721c211d8f8a9c58553524fdb9f5ab622ea742923afeefcb9cfd33837a054","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"2128deaaf0498a30dfaa48f7d19389dd3e2334fd0f7ff0f2639e3f4aa1a24345","test_name":"Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a1c200\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-9qv84v6w8w46\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d52026b5-448e-43ba-9476-e5eb16be630c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d52026b5-448e-43ba-9476-e5eb16be630c\",\n      \"name\": \"d52026b5-448e-43ba-9476-e5eb16be630c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:50:20.915935729Z\",\n      \"endTime\": \"2026-08-07T10:09:22.894129717Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a1c200\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-9qv84v6w8w46\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d52026b5-448e-43ba-9476-e5eb16be630c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d52026b5-448e-43ba-9476-e5eb16be630c\",\n      \"name\": \"d52026b5-448e-43ba-9476-e5eb16be630c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:50:20.915935729Z\",\n      \"endTime\": \"2026-08-07T10:09:22.894129717Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"c07aef81c0efa8477c98d7181ef78b51c52d4bf1a373e08c6cda5ad31472a188","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"417dfeffdd902a65bd5b4e92b2194352e1973c7abee4ebe0c561a040f49ebaa8","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-wznldc\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00110c160\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-wznldc: failed waiting for cluster=\"cluster-zstream-4-21-wznldc\" in resourcegroup=\"rg-zstream-upgrade-4-21-zdpthfv8zcvf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b4cbee7-cc43-4e89-8cda-6e9ed437b081\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b4cbee7-cc43-4e89-8cda-6e9ed437b081\",\n      \"name\": \"4b4cbee7-cc43-4e89-8cda-6e9ed437b081\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:16.551299537Z\",\n      \"endTime\": \"2026-08-07T10:30:34.394343748Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-wznldc\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00110c160\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-wznldc: failed waiting for cluster=\"cluster-zstream-4-21-wznldc\" in resourcegroup=\"rg-zstream-upgrade-4-21-zdpthfv8zcvf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b4cbee7-cc43-4e89-8cda-6e9ed437b081\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b4cbee7-cc43-4e89-8cda-6e9ed437b081\",\n      \"name\": \"4b4cbee7-cc43-4e89-8cda-6e9ed437b081\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:16.551299537Z\",\n      \"endTime\": \"2026-08-07T10:30:34.394343748Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"c61a99b96f0edf12174399d9f388e83b2f1190d018e4ad1543242be1eae930c2","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"b346cf052e2e32a3451598fccdcd5624d6ccd3d593f352963c3e2e563dac1ec7","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-hd9k7w\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f83560\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-hd9k7w: failed waiting for cluster=\"cluster-zstream-4-19-hd9k7w\" in resourcegroup=\"rg-zstream-upgrade-4-19-xkmpbjc2hppg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb960343-05a9-4903-8e0f-98f7b2b20dad\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb960343-05a9-4903-8e0f-98f7b2b20dad\",\n      \"name\": \"fb960343-05a9-4903-8e0f-98f7b2b20dad\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:33.741757841Z\",\n      \"endTime\": \"2026-08-07T10:30:34.774395519Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-hd9k7w\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f83560\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-hd9k7w: failed waiting for cluster=\"cluster-zstream-4-19-hd9k7w\" in resourcegroup=\"rg-zstream-upgrade-4-19-xkmpbjc2hppg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb960343-05a9-4903-8e0f-98f7b2b20dad\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fb960343-05a9-4903-8e0f-98f7b2b20dad\",\n      \"name\": \"fb960343-05a9-4903-8e0f-98f7b2b20dad\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:11:33.741757841Z\",\n      \"endTime\": \"2026-08-07T10:30:34.774395519Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"cffe6faba372d92411aaa9d1f21beb3c92948d4370409a8358f4a535d177f31d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"a4a58486b3a59fb37b023d0f3c5e9d3c795c024a9c780e78a484ca3c0c5a8186","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-g7dzzk with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000736780\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-g7dzzk: failed waiting for cluster=\"np-version-upgrade-cluster-g7dzzk\" in resourcegroup=\"rg-np-version-upgrade-g7dzzk-dsl78k9922jm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/594c0fa4-05f2-4ea4-8b5d-1a3135d56acd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/594c0fa4-05f2-4ea4-8b5d-1a3135d56acd\",\n      \"name\": \"594c0fa4-05f2-4ea4-8b5d-1a3135d56acd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:14:54.32628654Z\",\n      \"endTime\": \"2026-08-07T10:33:54.84317055Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-g7dzzk with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000736780\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-g7dzzk: failed waiting for cluster=\"np-version-upgrade-cluster-g7dzzk\" in resourcegroup=\"rg-np-version-upgrade-g7dzzk-dsl78k9922jm\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/594c0fa4-05f2-4ea4-8b5d-1a3135d56acd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/594c0fa4-05f2-4ea4-8b5d-1a3135d56acd\",\n      \"name\": \"594c0fa4-05f2-4ea4-8b5d-1a3135d56acd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:14:54.32628654Z\",\n      \"endTime\": \"2026-08-07T10:33:54.84317055Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d20d62248fc2b84a8119ea638cddb203f6da32a860eddfeb16a1aa2d622f3206","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"34da25653c9e294aba60c0baff88a413c5be08d506af1bad00fc9fe9dc16b817","test_name":"Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f5d360\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-7b7lgcj9lcp2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83c3d7e3-9f28-4b47-a149-0d9996540577\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83c3d7e3-9f28-4b47-a149-0d9996540577\",\n      \"name\": \"83c3d7e3-9f28-4b47-a149-0d9996540577\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:50.303149365Z\",\n      \"endTime\": \"2026-08-07T10:07:52.824880316Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f5d360\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-7b7lgcj9lcp2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83c3d7e3-9f28-4b47-a149-0d9996540577\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83c3d7e3-9f28-4b47-a149-0d9996540577\",\n      \"name\": \"83c3d7e3-9f28-4b47-a149-0d9996540577\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:50.303149365Z\",\n      \"endTime\": \"2026-08-07T10:07:52.824880316Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d46cfe9ef0635d1afa063424f3b9483da5bcdb7ac22ddc96437ecaf04a8b1c85","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"00d62c1b88a0797375914df008fcd53382cf1c8cdaca6621b70e3541ea6b30e9","test_name":"Customer should be able to use workload identity via the cluster OIDC issuer URL","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a150c0\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-sdkmrnl46j6f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf7b3b05-d460-4064-bc24-1a0f6977f6f2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf7b3b05-d460-4064-bc24-1a0f6977f6f2\",\n      \"name\": \"bf7b3b05-d460-4064-bc24-1a0f6977f6f2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:34:44.202796714Z\",\n      \"endTime\": \"2026-08-07T10:53:53.657736883Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000a150c0\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-sdkmrnl46j6f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf7b3b05-d460-4064-bc24-1a0f6977f6f2\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf7b3b05-d460-4064-bc24-1a0f6977f6f2\",\n      \"name\": \"bf7b3b05-d460-4064-bc24-1a0f6977f6f2\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:34:44.202796714Z\",\n      \"endTime\": \"2026-08-07T10:53:53.657736883Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"e89b5c10308790f6ca27d107a6e17d39b78a3babe46c899a7aea491345293404","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"12d98a2dd2486b4693a8e994b28bc05cbfe5418ebbd04805f10a5eeeca8ab11c","test_name":"Customer should be able to create an HCP cluster and custom node pool osDisk size","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000db71e0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-xrdfdg7kkplp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47cc38ff-0e8b-4173-bffc-3bc7b2d96c5e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47cc38ff-0e8b-4173-bffc-3bc7b2d96c5e\",\n      \"name\": \"47cc38ff-0e8b-4173-bffc-3bc7b2d96c5e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:58.960592588Z\",\n      \"endTime\": \"2026-08-07T10:08:02.875893645Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000db71e0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-xrdfdg7kkplp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47cc38ff-0e8b-4173-bffc-3bc7b2d96c5e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47cc38ff-0e8b-4173-bffc-3bc7b2d96c5e\",\n      \"name\": \"47cc38ff-0e8b-4173-bffc-3bc7b2d96c5e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:58.960592588Z\",\n      \"endTime\": \"2026-08-07T10:08:02.875893645Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"eb4b9089feb9eca7f3ba63d40cc8437ee8eb45e86617d3fc88a5a0890fedee6c","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"4eab6627e7d8a226e5291854ff24b9f665e13b20b82f505392fca8a5a0019436","test_name":"Customer should be able to create a HCP cluster and use cilium CNI plugin","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000aa90c0\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-k42bn2n5lrzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73ad444c-4e9e-41d9-aec1-9d6c5328d29b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73ad444c-4e9e-41d9-aec1-9d6c5328d29b\",\n      \"name\": \"73ad444c-4e9e-41d9-aec1-9d6c5328d29b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:38.807384796Z\",\n      \"endTime\": \"2026-08-07T10:07:42.691760944Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000aa90c0\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-k42bn2n5lrzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73ad444c-4e9e-41d9-aec1-9d6c5328d29b\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73ad444c-4e9e-41d9-aec1-9d6c5328d29b\",\n      \"name\": \"73ad444c-4e9e-41d9-aec1-9d6c5328d29b\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:48:38.807384796Z\",\n      \"endTime\": \"2026-08-07T10:07:42.691760944Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ee29a1dcea2400b7d66f88b00bc641616fecf4aae6c55bfa96cba1e370f3533b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"f32a67778d0d8771c50b5e72727683d2b6778132630fa22e6820b04c4a9db476","test_name":"Customer should not be able to perform various invalid operations on cluster resources","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00160a7e0\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rh692b5287x9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5f9205c-b39d-4c27-945b-4f80bbd75095\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5f9205c-b39d-4c27-945b-4f80bbd75095\",\n      \"name\": \"d5f9205c-b39d-4c27-945b-4f80bbd75095\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:34:43.806521563Z\",\n      \"endTime\": \"2026-08-07T10:53:43.63866572Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00160a7e0\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rh692b5287x9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5f9205c-b39d-4c27-945b-4f80bbd75095\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5f9205c-b39d-4c27-945b-4f80bbd75095\",\n      \"name\": \"d5f9205c-b39d-4c27-945b-4f80bbd75095\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:34:43.806521563Z\",\n      \"endTime\": \"2026-08-07T10:53:43.63866572Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"f1770343741649c368164045ebfd359e582cbcacf133b17d034be8dbc3e04b70","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"aa289a9977273eda5f25499128c8d5445097048089d0e40555cf7c25f1f29fcf","test_name":"Customer should be able to update node pool labels and taints","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001594360\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hzt9mx94mdk9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/451aa4a9-cd42-4973-8d90-f4fdfaba510c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/451aa4a9-cd42-4973-8d90-f4fdfaba510c\",\n      \"name\": \"451aa4a9-cd42-4973-8d90-f4fdfaba510c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:14:21.960226599Z\",\n      \"endTime\": \"2026-08-07T10:33:24.86845451Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc001594360\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hzt9mx94mdk9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/451aa4a9-cd42-4973-8d90-f4fdfaba510c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/451aa4a9-cd42-4973-8d90-f4fdfaba510c\",\n      \"name\": \"451aa4a9-cd42-4973-8d90-f4fdfaba510c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:14:21.960226599Z\",\n      \"endTime\": \"2026-08-07T10:33:24.86845451Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"f338e2e95e2fa57f122ebf750d8cad4fbdf06d1923c92c77cfa9e59fa1b2ebb8","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"da013a3bf7f3642bca9003f98895be727bdfa31ebb53e55021b618d8577cd9c7","test_name":"Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000900cc0\u003e: \n    failed to create HCP cluster np-downgrade-mf9lwq: failed waiting for cluster=\"np-downgrade-mf9lwq\" in resourcegroup=\"rg-np-downgrade-mf9lwq-glxf8hchwf7t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/371bd6e0-e117-436d-9fd1-2cf1185e74bd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/371bd6e0-e117-436d-9fd1-2cf1185e74bd\",\n      \"name\": \"371bd6e0-e117-436d-9fd1-2cf1185e74bd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:44.890366167Z\",\n      \"endTime\": \"2026-08-07T10:52:53.71768381Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000900cc0\u003e: \n    failed to create HCP cluster np-downgrade-mf9lwq: failed waiting for cluster=\"np-downgrade-mf9lwq\" in resourcegroup=\"rg-np-downgrade-mf9lwq-glxf8hchwf7t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/371bd6e0-e117-436d-9fd1-2cf1185e74bd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/371bd6e0-e117-436d-9fd1-2cf1185e74bd\",\n      \"name\": \"371bd6e0-e117-436d-9fd1-2cf1185e74bd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T10:33:44.890366167Z\",\n      \"endTime\": \"2026-08-07T10:52:53.71768381Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"fd8758c1db7c4e8e45bfc8dba22df74c56887d5c84d54e5cc77240c107aa2447","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085663875193114624","occurred_at":"2026-08-07T09:46:32Z","failed_at":"e2e","signature_id":"42468ebd1dfcf29874114f5fa8113739d93276a8a25b2c331bedbe9eca46fe43","test_name":"Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000eeac20\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-dsjbp4pttc2c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9be5fd-ee66-4839-963e-1cbe7f26ba5e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9be5fd-ee66-4839-963e-1cbe7f26ba5e\",\n      \"name\": \"5f9be5fd-ee66-4839-963e-1cbe7f26ba5e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:49:33.201712542Z\",\n      \"endTime\": \"2026-08-07T10:08:43.002606196Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000eeac20\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-dsjbp4pttc2c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9be5fd-ee66-4839-963e-1cbe7f26ba5e\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9be5fd-ee66-4839-963e-1cbe7f26ba5e\",\n      \"name\": \"5f9be5fd-ee66-4839-963e-1cbe7f26ba5e\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T09:49:33.201712542Z\",\n      \"endTime\": \"2026-08-07T10:08:43.002606196Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}}],"failure_pattern_summary":{"signature_count":44,"failure_pattern_ids":["0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b"],"matched_occurrences":44,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T07:22:18Z"},"failed_at":["e2e"],"failed_test_count":45,"occurrences":[{"row_id":"05299b0173b61382fbf13180bc3d0181552bc38fdced85d26526c4023e908f6e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"de9e064612ef52fbc8c9773477a956e0bc5e31d1b1ed58b3a0cc0cc3d0290e22","test_name":"Customer should create an HCP cluster and validate TLS certificates","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d77820\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-rfkspfpc9lm4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf567470-2457-474e-bfbf-426b4d3f4e89\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf567470-2457-474e-bfbf-426b4d3f4e89\",\n      \"name\": \"bf567470-2457-474e-bfbf-426b4d3f4e89\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:41:56.755602592Z\",\n      \"endTime\": \"2026-08-07T08:00:58.253066762Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d77820\u003e: \n    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-rfkspfpc9lm4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf567470-2457-474e-bfbf-426b4d3f4e89\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf567470-2457-474e-bfbf-426b4d3f4e89\",\n      \"name\": \"bf567470-2457-474e-bfbf-426b4d3f4e89\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:41:56.755602592Z\",\n      \"endTime\": \"2026-08-07T08:00:58.253066762Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0b53b27e84eae2276ca9e6ade8049b9d156108a0d92c7d01da0e6b7a808c456b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"f5d35358178602e808023f5909705f6e2976f7f51b2d8689d83141d70cb26006","test_name":"Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f134e0\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9clnhfsjnpjg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/324b144b-f8ab-4185-8bfa-af0f1a8e5d65\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/324b144b-f8ab-4185-8bfa-af0f1a8e5d65\",\n      \"name\": \"324b144b-f8ab-4185-8bfa-af0f1a8e5d65\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:52.832267773Z\",\n      \"endTime\": \"2026-08-07T08:23:58.982422749Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f134e0\u003e: \n    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9clnhfsjnpjg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/324b144b-f8ab-4185-8bfa-af0f1a8e5d65\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/324b144b-f8ab-4185-8bfa-af0f1a8e5d65\",\n      \"name\": \"324b144b-f8ab-4185-8bfa-af0f1a8e5d65\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:52.832267773Z\",\n      \"endTime\": \"2026-08-07T08:23:58.982422749Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0c8eb7c8754e582c723386ec023f144563765b49eb3e65021f4d4f46d75c1eb8","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"5f41eec5b630107ae74d4c3554976586c265bed58c792fcf02bc411b4db7e83c","test_name":"Customer should be able to create an HCP cluster and manage ImageDigestMirrors","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f04c60\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gk77sxfxkj7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c2a3023-08bd-477d-b9f0-db992626f6da\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c2a3023-08bd-477d-b9f0-db992626f6da\",\n      \"name\": \"3c2a3023-08bd-477d-b9f0-db992626f6da\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:27.410031703Z\",\n      \"endTime\": \"2026-08-07T08:23:28.97850991Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f04c60\u003e: \n    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gk77sxfxkj7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c2a3023-08bd-477d-b9f0-db992626f6da\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c2a3023-08bd-477d-b9f0-db992626f6da\",\n      \"name\": \"3c2a3023-08bd-477d-b9f0-db992626f6da\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:27.410031703Z\",\n      \"endTime\": \"2026-08-07T08:23:28.97850991Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0cadbf0bffce2c24eddde9bf9508c59947fb62efc973786cbdb32816208988e3","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"74fd191aaf7cead7463fe1516c12c673296e86a829f4b1a958b0a2ff45604ebe","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-h78jmf\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f8af80\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-h78jmf: failed waiting for cluster=\"cluster-zstream-4-22-h78jmf\" in resourcegroup=\"rg-zstream-upgrade-4-22-4vfs67mv7295\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/654a0ae4-a99e-44a0-a3c9-ee562e6f8ce7\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/654a0ae4-a99e-44a0-a3c9-ee562e6f8ce7\",\n      \"name\": \"654a0ae4-a99e-44a0-a3c9-ee562e6f8ce7\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:56.890514642Z\",\n      \"endTime\": \"2026-08-07T08:21:59.869736728Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-22-h78jmf\" with version 4.22.6 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f8af80\u003e: \n    failed to create HCP cluster cluster-zstream-4-22-h78jmf: failed waiting for cluster=\"cluster-zstream-4-22-h78jmf\" in resourcegroup=\"rg-zstream-upgrade-4-22-4vfs67mv7295\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/654a0ae4-a99e-44a0-a3c9-ee562e6f8ce7\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/654a0ae4-a99e-44a0-a3c9-ee562e6f8ce7\",\n      \"name\": \"654a0ae4-a99e-44a0-a3c9-ee562e6f8ce7\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:56.890514642Z\",\n      \"endTime\": \"2026-08-07T08:21:59.869736728Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0e69de1856eb3e3a1f7150db10fc0675d534b51b69e74413d8d03e7262a87afa","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"9dad2f2a15714594fc07b12a21efbb3c0861aadaf1dfcdde0ce3d7fb2feb0d69","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-mmxpgzrkljwg/cluster-candidate-4-21-bd5bgf should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006270e0\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-bd5bgf: failed waiting for cluster=\"cluster-candidate-4-21-bd5bgf\" in resourcegroup=\"rg-candidate-4-21-mmxpgzrkljwg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a50aefd6-7128-430d-b0ba-05385664ee09\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a50aefd6-7128-430d-b0ba-05385664ee09\",\n      \"name\": \"a50aefd6-7128-430d-b0ba-05385664ee09\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:44.159649344Z\",\n      \"endTime\": \"2026-08-07T08:21:50.285031425Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-21-mmxpgzrkljwg/cluster-candidate-4-21-bd5bgf should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006270e0\u003e: \n    failed to create HCP cluster cluster-candidate-4-21-bd5bgf: failed waiting for cluster=\"cluster-candidate-4-21-bd5bgf\" in resourcegroup=\"rg-candidate-4-21-mmxpgzrkljwg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a50aefd6-7128-430d-b0ba-05385664ee09\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a50aefd6-7128-430d-b0ba-05385664ee09\",\n      \"name\": \"a50aefd6-7128-430d-b0ba-05385664ee09\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:44.159649344Z\",\n      \"endTime\": \"2026-08-07T08:21:50.285031425Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"0fc34640a9036ff17cd184b4efe7cc8e05d5a8a969ecf8a77a6bea8ffe21b098","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"a2a9dc5f2e04f0e2bb7c7f5b65bc64e6de2c3af4b63395cd7e2f4fb714ef2be4","test_name":"FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000dcaf40\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-xkj87964wvc4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2b14a388-b91e-471d-b097-b1cdb23b1419\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000dcaf40\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-xkj87964wvc4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2b14a388-b91e-471d-b097-b1cdb23b1419\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f","failure_pattern":"ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: NoRegisteredProviderFound"}},{"row_id":"10f011d34a069ec800c1dab27d5f425504fde07b28614fdee25b91d4f37ebb2d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"35c703d17b434a3cdfc199e2bbb60a3c77b5eb3495497c699539afcf932e5ba3","test_name":"Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-zjzlzl\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004eae00\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-zjzlzl: failed waiting for cluster=\"cp-ystream-upgrade-4-21-zjzlzl\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-l85s99kdfn6d\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/040d3e9a-0653-42d4-8ba8-9d4c56c37b3f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/040d3e9a-0653-42d4-8ba8-9d4c56c37b3f\",\n      \"name\": \"040d3e9a-0653-42d4-8ba8-9d4c56c37b3f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:10.227916925Z\",\n      \"endTime\": \"2026-08-07T08:22:19.061448954Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster \"cp-ystream-upgrade-4-21-zjzlzl\" with install version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0004eae00\u003e: \n    failed to create HCP cluster cp-ystream-upgrade-4-21-zjzlzl: failed waiting for cluster=\"cp-ystream-upgrade-4-21-zjzlzl\" in resourcegroup=\"rg-cp-ystream-upgrade-4-21-l85s99kdfn6d\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/040d3e9a-0653-42d4-8ba8-9d4c56c37b3f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/040d3e9a-0653-42d4-8ba8-9d4c56c37b3f\",\n      \"name\": \"040d3e9a-0653-42d4-8ba8-9d4c56c37b3f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:10.227916925Z\",\n      \"endTime\": \"2026-08-07T08:22:19.061448954Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"24409b0dc5095e3b84c4908742b8100cd13fcc1b065235016867708af59fd596","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"9210fd6c229792098ceaba0620c59c9bc7b2fdd4049e127e9fa211f4967c8e88","test_name":"Authorized CIDRs Connectivity should allow API access only from authorized VM IP","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f90da0\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-vbs8sjnthcgf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01c94751-0c56-4bb9-b30b-8b088dcc51eb\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01c94751-0c56-4bb9-b30b-8b088dcc51eb\",\n      \"name\": \"01c94751-0c56-4bb9-b30b-8b088dcc51eb\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:40:39.149843476Z\",\n      \"endTime\": \"2026-08-07T07:59:48.346334102Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster \"cidr-connectivity-test\" with authorized CIDRs\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f90da0\u003e: \n    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-vbs8sjnthcgf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01c94751-0c56-4bb9-b30b-8b088dcc51eb\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01c94751-0c56-4bb9-b30b-8b088dcc51eb\",\n      \"name\": \"01c94751-0c56-4bb9-b30b-8b088dcc51eb\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:40:39.149843476Z\",\n      \"endTime\": \"2026-08-07T07:59:48.346334102Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"257478cee1f249fb0375ab0ebf47362b36eeea696ce6c47210f91a8bbfcfd74d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"db22d3f63476044a851bb065aa2f2862b371c4743f70defc5fe4829d132b7d7a","test_name":"Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc001516810\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b798a1e-bf58-47be-b8ba-654a2b4d6825\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b798a1e-bf58-47be-b8ba-654a2b4d6825\",\n      \"name\": \"4b798a1e-bf58-47be-b8ba-654a2b4d6825\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:40:45.599532342Z\",\n      \"endTime\": \"2026-08-07T07:59:48.345719335Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster \"basic-hcp-cluster\" to complete creation (timeout '20.000000' minutes)\nUnexpected error:\n    \u003c*exported.ResponseError | 0xc001516810\u003e: \n    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b798a1e-bf58-47be-b8ba-654a2b4d6825\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4b798a1e-bf58-47be-b8ba-654a2b4d6825\",\n      \"name\": \"4b798a1e-bf58-47be-b8ba-654a2b4d6825\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:40:45.599532342Z\",\n      \"endTime\": \"2026-08-07T07:59:48.345719335Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"283f3ea3b7651b8f8b22cd12dd6361e4f8a5438bb4c5041c96b25b1d9c16aa60","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"de9cd6c807d85dc17c9beea19134b276126edbcf359555678cc4e31d59fc2884","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-2h5fgg with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000fcd3c0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-2h5fgg: failed waiting for cluster=\"np-version-upgrade-cluster-2h5fgg\" in resourcegroup=\"rg-np-version-upgrade-2h5fgg-z5c2fzfh7shh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/725f2e9d-bfd0-4685-9daa-9b361164f5b0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/725f2e9d-bfd0-4685-9daa-9b361164f5b0\",\n      \"name\": \"725f2e9d-bfd0-4685-9daa-9b361164f5b0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:07:33.440164449Z\",\n      \"endTime\": \"2026-08-07T08:26:39.186830239Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-2h5fgg with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000fcd3c0\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-2h5fgg: failed waiting for cluster=\"np-version-upgrade-cluster-2h5fgg\" in resourcegroup=\"rg-np-version-upgrade-2h5fgg-z5c2fzfh7shh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/725f2e9d-bfd0-4685-9daa-9b361164f5b0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/725f2e9d-bfd0-4685-9daa-9b361164f5b0\",\n      \"name\": \"725f2e9d-bfd0-4685-9daa-9b361164f5b0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:07:33.440164449Z\",\n      \"endTime\": \"2026-08-07T08:26:39.186830239Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"2c88f32aefd23db0d90f1bf69b0f0c08cc31b9a695fa5ab3c8c4cf7bf6706bdc","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"b03f48795569ce433282b46a0d39585a64fb185ae56e9c8fa2c0e195e219c4c8","test_name":"Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000158ec0\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-d5nhvd54wsrw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/738a7a6e-9b1d-4fdc-b0d5-2db6b0580aea\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/738a7a6e-9b1d-4fdc-b0d5-2db6b0580aea\",\n      \"name\": \"738a7a6e-9b1d-4fdc-b0d5-2db6b0580aea\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:45.221385702Z\",\n      \"endTime\": \"2026-08-07T07:59:18.318655469Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster \"autoscaling-hcp-cluster\" with custom autoscaling\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000158ec0\u003e: \n    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-d5nhvd54wsrw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/738a7a6e-9b1d-4fdc-b0d5-2db6b0580aea\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/738a7a6e-9b1d-4fdc-b0d5-2db6b0580aea\",\n      \"name\": \"738a7a6e-9b1d-4fdc-b0d5-2db6b0580aea\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:45.221385702Z\",\n      \"endTime\": \"2026-08-07T07:59:18.318655469Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"310689c8267b70f87bec07416f7a362e0051b87215e2f7072c9b4b29167d45ea","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"9ae2354ef5634581314252053e50e7777c0606efb00925edb028af2730ad76f1","test_name":"Customer should be able to lifecycle and confirm external auth on a cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000aa0fc0\u003e: \n    failed to create HCP cluster ea-list69n8tf: failed waiting for cluster=\"ea-list69n8tf\" in resourcegroup=\"ea-list-75r9x976dxwp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2984486-425c-4a9d-9aea-9ce5bec715f3\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2984486-425c-4a9d-9aea-9ce5bec715f3\",\n      \"name\": \"c2984486-425c-4a9d-9aea-9ce5bec715f3\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:23.590561215Z\",\n      \"endTime\": \"2026-08-07T08:22:29.129478255Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000aa0fc0\u003e: \n    failed to create HCP cluster ea-list69n8tf: failed waiting for cluster=\"ea-list69n8tf\" in resourcegroup=\"ea-list-75r9x976dxwp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2984486-425c-4a9d-9aea-9ce5bec715f3\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2984486-425c-4a9d-9aea-9ce5bec715f3\",\n      \"name\": \"c2984486-425c-4a9d-9aea-9ce5bec715f3\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:23.590561215Z\",\n      \"endTime\": \"2026-08-07T08:22:29.129478255Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"35e97738665e4bc858e8b0556538cd203ba4314c22cefa1cd24409e349e5f7cf","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"2936b0a37a9493678b414857f9bb365fbe59f7e7b947efd786736f4863b5f1e0","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-npgwtj\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000dfaee0\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-npgwtj: failed waiting for cluster=\"cluster-zstream-4-19-npgwtj\" in resourcegroup=\"rg-zstream-upgrade-4-19-v7vhrw4pl2vw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2cc2b4c-0d35-419d-aa29-305904f96dbf\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2cc2b4c-0d35-419d-aa29-305904f96dbf\",\n      \"name\": \"c2cc2b4c-0d35-419d-aa29-305904f96dbf\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:07.682896221Z\",\n      \"endTime\": \"2026-08-07T08:22:09.191420175Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-19-npgwtj\" with version 4.19.39 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000dfaee0\u003e: \n    failed to create HCP cluster cluster-zstream-4-19-npgwtj: failed waiting for cluster=\"cluster-zstream-4-19-npgwtj\" in resourcegroup=\"rg-zstream-upgrade-4-19-v7vhrw4pl2vw\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2cc2b4c-0d35-419d-aa29-305904f96dbf\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2cc2b4c-0d35-419d-aa29-305904f96dbf\",\n      \"name\": \"c2cc2b4c-0d35-419d-aa29-305904f96dbf\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:07.682896221Z\",\n      \"endTime\": \"2026-08-07T08:22:09.191420175Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"3c1dcd68c1f50464fea8cc6c090d1276f35f0ab4bb67f742541c1e42971fdeaf","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"28d34f39a752f09909b66036c5755b61dfd94a1dce7afa2218622cfd003f0351","test_name":"Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ea51e0\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-bwq249cs7nmj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cfd5160-efbb-4871-94e8-27b04cc12b27\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cfd5160-efbb-4871-94e8-27b04cc12b27\",\n      \"name\": \"6cfd5160-efbb-4871-94e8-27b04cc12b27\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:44:25.032635074Z\",\n      \"endTime\": \"2026-08-07T08:03:28.635414967Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ea51e0\u003e: \n    failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-bwq249cs7nmj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cfd5160-efbb-4871-94e8-27b04cc12b27\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6cfd5160-efbb-4871-94e8-27b04cc12b27\",\n      \"name\": \"6cfd5160-efbb-4871-94e8-27b04cc12b27\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:44:25.032635074Z\",\n      \"endTime\": \"2026-08-07T08:03:28.635414967Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"490f2823e9e733b2bc0192fad41d39017d3b3df9c98ef3b10944915d799694d4","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"b739387933134b17c9326db592dd3453a19d55a5a9130d082d33e05202b777fb","test_name":"Customer should be able to create an HCP cluster and custom node pool osDisk size","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00087a7e0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-792swr5j7rgq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d2a790b-82c6-4f73-9ae2-27326edcb0b5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d2a790b-82c6-4f73-9ae2-27326edcb0b5\",\n      \"name\": \"1d2a790b-82c6-4f73-9ae2-27326edcb0b5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:42.024129132Z\",\n      \"endTime\": \"2026-08-07T07:58:48.375392766Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster \"hcp-cluster-np-128\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00087a7e0\u003e: \n    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-792swr5j7rgq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d2a790b-82c6-4f73-9ae2-27326edcb0b5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d2a790b-82c6-4f73-9ae2-27326edcb0b5\",\n      \"name\": \"1d2a790b-82c6-4f73-9ae2-27326edcb0b5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:42.024129132Z\",\n      \"endTime\": \"2026-08-07T07:58:48.375392766Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4c097df301879ddf60fa0a34f26d40e2467ccbf725ab5ef406802f529dcdfa22","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"86f5e68a3fe7665365f3401512d9c19ac1f225fde6fd9deda3b6f1fcfa7197cd","test_name":"HCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e03020\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-wtr4hdzs6sm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f38768a-9e11-4241-8c9b-2323d6f72f2f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f38768a-9e11-4241-8c9b-2323d6f72f2f\",\n      \"name\": \"8f38768a-9e11-4241-8c9b-2323d6f72f2f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:08.824762741Z\",\n      \"endTime\": \"2026-08-07T08:22:09.045166413Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:81]: failed to create HCP cluster for GPU nodepool test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e03020\u003e: \n    failed to create HCP cluster cluster-gpu-np: failed waiting for cluster=\"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-wtr4hdzs6sm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f38768a-9e11-4241-8c9b-2323d6f72f2f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8f38768a-9e11-4241-8c9b-2323d6f72f2f\",\n      \"name\": \"8f38768a-9e11-4241-8c9b-2323d6f72f2f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:08.824762741Z\",\n      \"endTime\": \"2026-08-07T08:22:09.045166413Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"4c6c46f7bd812cc440d0d2727f70748e24669809ea825db654b4ac17cf513d66","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"14c80964b9874c9081110be783c8e717ab9db60e5a66ec45327b015be13e84aa","test_name":"Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed\nfail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed","failure_pattern_match":{"status":"clustered","failure_pattern_id":"e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b","failure_pattern":"Cluster provisioning failed","search_query":"Cluster provisioning failed"}},{"row_id":"62328a7db059647836205f6c24352cd64d4f9fbc9b1c3962ac898c870f854d01","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"d20dff25b1077f154d6430a2a4a277bc7bc842ff37e2b66c8036af5a51bc6a97","test_name":"Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0014884e0\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-9xflhr88sxpq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f787530f-5e0f-4f43-b4d5-05752bffea20\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f787530f-5e0f-4f43-b4d5-05752bffea20\",\n      \"name\": \"f787530f-5e0f-4f43-b4d5-05752bffea20\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:42.131709982Z\",\n      \"endTime\": \"2026-08-07T08:23:48.958799388Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: failed to create HCP cluster ephemeral-disk\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0014884e0\u003e: \n    failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-9xflhr88sxpq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f787530f-5e0f-4f43-b4d5-05752bffea20\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f787530f-5e0f-4f43-b4d5-05752bffea20\",\n      \"name\": \"f787530f-5e0f-4f43-b4d5-05752bffea20\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:42.131709982Z\",\n      \"endTime\": \"2026-08-07T08:23:48.958799388Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"641044319c52ac53d96d4665801be01736841f8efcf32f790324b7e9fee8156a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"30338ef2d47949f5e9b6aaebf23fa70a98576ce72bd7f3ec806a5655d72c1fc7","test_name":"Customer should be able to create node pools with ARM64-based VMs","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00099a980\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-g8cbmtj6mmr9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b8e9cfab-13a6-48e3-9328-7e39cfb49034\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b8e9cfab-13a6-48e3-9328-7e39cfb49034\",\n      \"name\": \"b8e9cfab-13a6-48e3-9328-7e39cfb49034\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:45.500543412Z\",\n      \"endTime\": \"2026-08-07T07:58:48.426146864Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster \"arm64-vm-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00099a980\u003e: \n    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-g8cbmtj6mmr9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b8e9cfab-13a6-48e3-9328-7e39cfb49034\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b8e9cfab-13a6-48e3-9328-7e39cfb49034\",\n      \"name\": \"b8e9cfab-13a6-48e3-9328-7e39cfb49034\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:45.500543412Z\",\n      \"endTime\": \"2026-08-07T07:58:48.426146864Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"7497b7de24c13ea57b683ccb3cfb13a5ac0fa3d5bd3f42eedd5d443f4a641f4b","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"7835149125518536e05b9a836ba114beb73af29f853c6e274b60c9c525fe4553","test_name":"Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0012be800\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-4mfdlgp8n4sd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a31752b-9506-4ffa-869a-b87e9c298d7d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a31752b-9506-4ffa-869a-b87e9c298d7d\",\n      \"name\": \"4a31752b-9506-4ffa-869a-b87e9c298d7d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:41:26.204906756Z\",\n      \"endTime\": \"2026-08-07T08:00:28.286763657Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster \"cilium-cluster\" with no CNI and private etcd\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0012be800\u003e: \n    failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-4mfdlgp8n4sd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a31752b-9506-4ffa-869a-b87e9c298d7d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4a31752b-9506-4ffa-869a-b87e9c298d7d\",\n      \"name\": \"4a31752b-9506-4ffa-869a-b87e9c298d7d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:41:26.204906756Z\",\n      \"endTime\": \"2026-08-07T08:00:28.286763657Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"78893b39678b5748eb4ba13ca8aa501ad5ca868f25f89ee238fa6333e4abbcfa","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"ffd3af46b5b3eb0e0dbdf0b633687c2731a794adabd3de45a85d7d40c39569ee","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-npqbdrvg46ff/cluster-candidate-4-20-lx8msx should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000c07380\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-lx8msx: failed waiting for cluster=\"cluster-candidate-4-20-lx8msx\" in resourcegroup=\"rg-candidate-4-20-npqbdrvg46ff\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b777475c-0ab5-40d8-9d03-7bf29b62c033\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b777475c-0ab5-40d8-9d03-7bf29b62c033\",\n      \"name\": \"b777475c-0ab5-40d8-9d03-7bf29b62c033\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:43.670313994Z\",\n      \"endTime\": \"2026-08-07T08:21:49.838672232Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-20-npqbdrvg46ff/cluster-candidate-4-20-lx8msx should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000c07380\u003e: \n    failed to create HCP cluster cluster-candidate-4-20-lx8msx: failed waiting for cluster=\"cluster-candidate-4-20-lx8msx\" in resourcegroup=\"rg-candidate-4-20-npqbdrvg46ff\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b777475c-0ab5-40d8-9d03-7bf29b62c033\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b777475c-0ab5-40d8-9d03-7bf29b62c033\",\n      \"name\": \"b777475c-0ab5-40d8-9d03-7bf29b62c033\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:43.670313994Z\",\n      \"endTime\": \"2026-08-07T08:21:49.838672232Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"7ae060a8525ad44a6a391539f16f2080dbf28ad999d20a8698cafff9a499c5e8","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"451fe8002ad81e914470397179df66df2a60819e7d3f75f884096bac40e2f32b","test_name":"Customer should not be able to perform various invalid operations on cluster resources","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006178c0\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-97w9f2h6wgzh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7696fd9c-9de9-471f-9075-71504c9d46ae\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7696fd9c-9de9-471f-9075-71504c9d46ae\",\n      \"name\": \"7696fd9c-9de9-471f-9075-71504c9d46ae\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:26:32.659885197Z\",\n      \"endTime\": \"2026-08-07T08:45:38.228821312Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006178c0\u003e: \n    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-97w9f2h6wgzh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7696fd9c-9de9-471f-9075-71504c9d46ae\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7696fd9c-9de9-471f-9075-71504c9d46ae\",\n      \"name\": \"7696fd9c-9de9-471f-9075-71504c9d46ae\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:26:32.659885197Z\",\n      \"endTime\": \"2026-08-07T08:45:38.228821312Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"874f0204ba647b9148460c5bed6bf7d3b818087c97b655d57554463f8f83b7a3","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"495dc21fbdf026e024531a3ffb5a26782b5dbd41f960983bc4aefe5aa16b093b","test_name":"Customer should be able to update node pool labels and taints","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b09340\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-rlth2nsfvv2s\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5f17bd6-58e5-4000-8148-6baacfbbc8a0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5f17bd6-58e5-4000-8148-6baacfbbc8a0\",\n      \"name\": \"f5f17bd6-58e5-4000-8148-6baacfbbc8a0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:07:01.226494096Z\",\n      \"endTime\": \"2026-08-07T08:26:09.148107822Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b09340\u003e: \n    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-rlth2nsfvv2s\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5f17bd6-58e5-4000-8148-6baacfbbc8a0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f5f17bd6-58e5-4000-8148-6baacfbbc8a0\",\n      \"name\": \"f5f17bd6-58e5-4000-8148-6baacfbbc8a0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:07:01.226494096Z\",\n      \"endTime\": \"2026-08-07T08:26:09.148107822Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"8765f4148acb3b5fa5790759a81e1a23b1d7975570a500cba381582a39485f29","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"cf5a2b0b5bd2e032d84007e2bd2b6c39b63d2c3376264447a8cf00f49785de68","test_name":"Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005fd1c0\u003e: \n    failed to create HCP cluster np-dg-minor-rz2wmd: failed waiting for cluster=\"np-dg-minor-rz2wmd\" in resourcegroup=\"rg-np-dg-minor-rz2wmd-znz4fjqdj58z\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faa57d65-680f-42de-a519-86b1440cd569\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faa57d65-680f-42de-a519-86b1440cd569\",\n      \"name\": \"faa57d65-680f-42de-a519-86b1440cd569\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:26:00.531037933Z\",\n      \"endTime\": \"2026-08-07T08:45:08.226547246Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:741]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005fd1c0\u003e: \n    failed to create HCP cluster np-dg-minor-rz2wmd: failed waiting for cluster=\"np-dg-minor-rz2wmd\" in resourcegroup=\"rg-np-dg-minor-rz2wmd-znz4fjqdj58z\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faa57d65-680f-42de-a519-86b1440cd569\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/faa57d65-680f-42de-a519-86b1440cd569\",\n      \"name\": \"faa57d65-680f-42de-a519-86b1440cd569\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:26:00.531037933Z\",\n      \"endTime\": \"2026-08-07T08:45:08.226547246Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"87da03bae67eea915e4f0c51f1b9b80781d66843b60a5609980d791c18c2225d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"83bbef9e9b345a98b7bd86843c4a0bbd1a12a9e52683d09366a67392ca16ca1f","test_name":"Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d04bc0\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-wdxxbqlz6m5h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/da649013-c972-4cef-b9f6-4ff38b1a8541\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/da649013-c972-4cef-b9f6-4ff38b1a8541\",\n      \"name\": \"da649013-c972-4cef-b9f6-4ff38b1a8541\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:41:26.640781162Z\",\n      \"endTime\": \"2026-08-07T08:00:28.270790469Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster \"private-kv-cluster\" with private keyvault\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000d04bc0\u003e: \n    failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-wdxxbqlz6m5h\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/da649013-c972-4cef-b9f6-4ff38b1a8541\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/da649013-c972-4cef-b9f6-4ff38b1a8541\",\n      \"name\": \"da649013-c972-4cef-b9f6-4ff38b1a8541\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:41:26.640781162Z\",\n      \"endTime\": \"2026-08-07T08:00:28.270790469Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"8a0f80d51ee0279d2d8ad1ccbba229884e4869ef474ae34e1afa2751649ff3bd","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"67057b5012d86ce1cdd8d52ce8e4cb6dcbec58e6f25262233655b83c453f88bd","test_name":"Customer should be able to create an HCP cluster and manage pull secrets","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000edede0\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rr54vcdmhllh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/975bbe48-9ad2-4f40-9017-c758210caccd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/975bbe48-9ad2-4f40-9017-c758210caccd\",\n      \"name\": \"975bbe48-9ad2-4f40-9017-c758210caccd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:44.920320394Z\",\n      \"endTime\": \"2026-08-07T07:58:48.458055521Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000edede0\u003e: \n    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rr54vcdmhllh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/975bbe48-9ad2-4f40-9017-c758210caccd\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/975bbe48-9ad2-4f40-9017-c758210caccd\",\n      \"name\": \"975bbe48-9ad2-4f40-9017-c758210caccd\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:44.920320394Z\",\n      \"endTime\": \"2026-08-07T07:58:48.458055521Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"8a6b3f757721cc378da81e51ea2dbee94f527290831bd6d7b21a7175373b8fb0","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"7f10459aab3e30729a1861e5489739968b95f475506a8fe1e19fd42f76e03d80","test_name":"Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008f9ac0\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-hvvbzsv7d89t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d57fcbc-56f3-444d-8feb-c59fc65e435d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d57fcbc-56f3-444d-8feb-c59fc65e435d\",\n      \"name\": \"8d57fcbc-56f3-444d-8feb-c59fc65e435d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:40:38.476240936Z\",\n      \"endTime\": \"2026-08-07T07:59:38.253381812Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster \"private-kas\" with private KAS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0008f9ac0\u003e: \n    failed to create HCP cluster private-kas: failed waiting for cluster=\"private-kas\" in resourcegroup=\"private-kas-hvvbzsv7d89t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d57fcbc-56f3-444d-8feb-c59fc65e435d\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d57fcbc-56f3-444d-8feb-c59fc65e435d\",\n      \"name\": \"8d57fcbc-56f3-444d-8feb-c59fc65e435d\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:40:38.476240936Z\",\n      \"endTime\": \"2026-08-07T07:59:38.253381812Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"8ff845f5ddf49f538d5e0c073d7e283fc27deda4019c2e92026613f451a6bbeb","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"0e6c677b1a1d11de6fe7833aafdf9149e03f90db8d34118fdfec5f1228c9c51c","test_name":"Customer should be able to create an HCP cluster with Image Registry not present","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010c4500\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-45mbbp2z75zs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aeafe766-4230-4fa0-bacd-c1249d6202c7\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aeafe766-4230-4fa0-bacd-c1249d6202c7\",\n      \"name\": \"aeafe766-4230-4fa0-bacd-c1249d6202c7\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:38.156298949Z\",\n      \"endTime\": \"2026-08-07T08:23:38.988224753Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010c4500\u003e: \n    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-45mbbp2z75zs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aeafe766-4230-4fa0-bacd-c1249d6202c7\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aeafe766-4230-4fa0-bacd-c1249d6202c7\",\n      \"name\": \"aeafe766-4230-4fa0-bacd-c1249d6202c7\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:38.156298949Z\",\n      \"endTime\": \"2026-08-07T08:23:38.988224753Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"94c4c054c94cdc9bce42e3c570e058307dc1ae42a74014176a25e2d0a3163b6e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"604160db0c2dd59b4722afdfaa73ac0064b49dcd3b7fe94295dd3c782113df7d","test_name":"Customer should be able to use workload identity via the cluster OIDC issuer URL","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0014d6fc0\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-75dwb2g2jtz8\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6372cdab-1153-4a6c-83a7-7fc18031bc6c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6372cdab-1153-4a6c-83a7-7fc18031bc6c\",\n      \"name\": \"6372cdab-1153-4a6c-83a7-7fc18031bc6c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:26:02.334507694Z\",\n      \"endTime\": \"2026-08-07T08:45:08.231237991Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0014d6fc0\u003e: \n    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-75dwb2g2jtz8\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6372cdab-1153-4a6c-83a7-7fc18031bc6c\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6372cdab-1153-4a6c-83a7-7fc18031bc6c\",\n      \"name\": \"6372cdab-1153-4a6c-83a7-7fc18031bc6c\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:26:02.334507694Z\",\n      \"endTime\": \"2026-08-07T08:45:08.231237991Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"a0fbe741522f11d7297c32623944278e3e4d408ed3e46b568e4ae1f81d2897d6","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"51e2dd5c5028ffbf420fe95e777d906128608f82be65f3a579b906e6b9118626","test_name":"Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-9944f8 with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00101a640\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-9944f8: failed waiting for cluster=\"np-version-upgrade-cluster-9944f8\" in resourcegroup=\"rg-np-version-upgrade-9944f8-chq29tdz87tg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bb3d58fb-06ea-49da-a30f-d4842bcbbb45\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bb3d58fb-06ea-49da-a30f-d4842bcbbb45\",\n      \"name\": \"bb3d58fb-06ea-49da-a30f-d4842bcbbb45\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:24:27.289872344Z\",\n      \"endTime\": \"2026-08-07T08:43:30.114142692Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-9944f8 with version 4.21.28\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc00101a640\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-9944f8: failed waiting for cluster=\"np-version-upgrade-cluster-9944f8\" in resourcegroup=\"rg-np-version-upgrade-9944f8-chq29tdz87tg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bb3d58fb-06ea-49da-a30f-d4842bcbbb45\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bb3d58fb-06ea-49da-a30f-d4842bcbbb45\",\n      \"name\": \"bb3d58fb-06ea-49da-a30f-d4842bcbbb45\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:24:27.289872344Z\",\n      \"endTime\": \"2026-08-07T08:43:30.114142692Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"a197c449ec51648e5f82d1a4f69158adb0d3e3e0cef799881369ce229c11bf1a","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"2b65298fd83d55b940b4663b25862d3af888466db21f65fde107f64706610d13","test_name":"Customer should be able to create a cluster with an external auth config and get the external auth config","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b04a00\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-vlvwkrpx2v92\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12976d97-15d5-415e-91a9-9591e2ffa94f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12976d97-15d5-415e-91a9-9591e2ffa94f\",\n      \"name\": \"12976d97-15d5-415e-91a9-9591e2ffa94f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:22.661330017Z\",\n      \"endTime\": \"2026-08-07T08:22:28.981972045Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000b04a00\u003e: \n    failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-vlvwkrpx2v92\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12976d97-15d5-415e-91a9-9591e2ffa94f\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/12976d97-15d5-415e-91a9-9591e2ffa94f\",\n      \"name\": \"12976d97-15d5-415e-91a9-9591e2ffa94f\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:03:22.661330017Z\",\n      \"endTime\": \"2026-08-07T08:22:28.981972045Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"a3f9f60288040e7c4f72d4da95f8056f85593ba194a8145c576d0b4651a1bac5","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"76070388c0b8d6ec1ebf419a3bacae3389450b10b08be31c7a935da227f5a1c6","test_name":"Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-b5lkzn at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f24fa0\u003e: \n    failed to create HCP cluster np-skip-minor-b5lkzn: failed waiting for cluster=\"np-skip-minor-b5lkzn\" in resourcegroup=\"rg-np-skip-minor-b5lkzn-tctmpt8792nt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3089f671-a356-4be4-b07c-acfd4a059332\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3089f671-a356-4be4-b07c-acfd4a059332\",\n      \"name\": \"3089f671-a356-4be4-b07c-acfd4a059332\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:11.531342418Z\",\n      \"endTime\": \"2026-08-07T08:44:18.198700114Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:462]: failed to create HCP cluster np-skip-minor-b5lkzn at version 4.22.9\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f24fa0\u003e: \n    failed to create HCP cluster np-skip-minor-b5lkzn: failed waiting for cluster=\"np-skip-minor-b5lkzn\" in resourcegroup=\"rg-np-skip-minor-b5lkzn-tctmpt8792nt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3089f671-a356-4be4-b07c-acfd4a059332\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3089f671-a356-4be4-b07c-acfd4a059332\",\n      \"name\": \"3089f671-a356-4be4-b07c-acfd4a059332\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:11.531342418Z\",\n      \"endTime\": \"2026-08-07T08:44:18.198700114Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"ae8c3e927555935cbb94cffb1243105ff99f0d1e65318ce54bcbc3f57db161d4","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"14cd6c6a937f4a9da39011576fa92c2d28e84666d80503bee3cc7e6d935d6aef","test_name":"Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-c84mcm\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000711ac0\u003e: \n    failed to create HCP cluster np-noedge-c84mcm: failed waiting for cluster=\"np-noedge-c84mcm\" in resourcegroup=\"rg-np-noedge-c84mcm-sld9xb9xgfv9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/041c18b4-8af2-4f6b-9c15-bc3fc0c71ab0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/041c18b4-8af2-4f6b-9c15-bc3fc0c71ab0\",\n      \"name\": \"041c18b4-8af2-4f6b-9c15-bc3fc0c71ab0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:11.802871441Z\",\n      \"endTime\": \"2026-08-07T08:44:18.201593337Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-c84mcm\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000711ac0\u003e: \n    failed to create HCP cluster np-noedge-c84mcm: failed waiting for cluster=\"np-noedge-c84mcm\" in resourcegroup=\"rg-np-noedge-c84mcm-sld9xb9xgfv9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/041c18b4-8af2-4f6b-9c15-bc3fc0c71ab0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/041c18b4-8af2-4f6b-9c15-bc3fc0c71ab0\",\n      \"name\": \"041c18b4-8af2-4f6b-9c15-bc3fc0c71ab0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:11.802871441Z\",\n      \"endTime\": \"2026-08-07T08:44:18.201593337Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"b44fbad7a54f4dc74b25878bfe7dd9a1acb722babc14fb71cc57bf4e6326b058","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"95aabd7261a84f5b46bb264eb8e4800c9bfbe3117360bc511ace2d3821b74dc6","test_name":"Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000521980\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-scdf9m8k79k9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0157cada-978a-45b0-958b-656bdab977a1\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0157cada-978a-45b0-958b-656bdab977a1\",\n      \"name\": \"0157cada-978a-45b0-958b-656bdab977a1\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:45:09.110824254Z\",\n      \"endTime\": \"2026-08-07T08:04:18.66846209Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000521980\u003e: \n    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-scdf9m8k79k9\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0157cada-978a-45b0-958b-656bdab977a1\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0157cada-978a-45b0-958b-656bdab977a1\",\n      \"name\": \"0157cada-978a-45b0-958b-656bdab977a1\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:45:09.110824254Z\",\n      \"endTime\": \"2026-08-07T08:04:18.66846209Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"b6c4b23d65fd3342c6998ceba91337d5a5b07c1ddbb0570bdc442d0cef91ee75","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"970cf2f6cc585f3d41c507b6c4e84d5f8917a5bc8290f1212efd92ccbf8107c7","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-bg9c5b\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005aaa40\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-bg9c5b: failed waiting for cluster=\"cluster-zstream-4-21-bg9c5b\" in resourcegroup=\"rg-zstream-upgrade-4-21-5w7nht69694p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7bc3758a-59a6-4959-a689-4b91220de404\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7bc3758a-59a6-4959-a689-4b91220de404\",\n      \"name\": \"7bc3758a-59a6-4959-a689-4b91220de404\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:46.515996946Z\",\n      \"endTime\": \"2026-08-07T08:21:49.485807276Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-21-bg9c5b\" with version 4.21.27 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005aaa40\u003e: \n    failed to create HCP cluster cluster-zstream-4-21-bg9c5b: failed waiting for cluster=\"cluster-zstream-4-21-bg9c5b\" in resourcegroup=\"rg-zstream-upgrade-4-21-5w7nht69694p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7bc3758a-59a6-4959-a689-4b91220de404\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7bc3758a-59a6-4959-a689-4b91220de404\",\n      \"name\": \"7bc3758a-59a6-4959-a689-4b91220de404\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:46.515996946Z\",\n      \"endTime\": \"2026-08-07T08:21:49.485807276Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d09008c11d1ce9cdb12b5f1f97812c65c41b244d6b4628942a459613a866242d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"8b503fab1ff3eba8246a81add6e26656cf4c8be81b764481eeaaa07145d8a784","test_name":"Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-dfnt74 with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f32940\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-dfnt74: failed waiting for cluster=\"np-version-upgrade-cluster-dfnt74\" in resourcegroup=\"rg-np-version-upgrade-dfnt74-lmcscg77nkpk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f854e09d-3fb1-47f1-b7ba-7fa657c8c3c9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f854e09d-3fb1-47f1-b7ba-7fa657c8c3c9\",\n      \"name\": \"f854e09d-3fb1-47f1-b7ba-7fa657c8c3c9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:27.919908174Z\",\n      \"endTime\": \"2026-08-07T08:44:28.270170603Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:133]: failed to create HCP cluster np-version-upgrade-cluster-dfnt74 with version 4.20.32\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000f32940\u003e: \n    failed to create HCP cluster np-version-upgrade-cluster-dfnt74: failed waiting for cluster=\"np-version-upgrade-cluster-dfnt74\" in resourcegroup=\"rg-np-version-upgrade-dfnt74-lmcscg77nkpk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f854e09d-3fb1-47f1-b7ba-7fa657c8c3c9\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f854e09d-3fb1-47f1-b7ba-7fa657c8c3c9\",\n      \"name\": \"f854e09d-3fb1-47f1-b7ba-7fa657c8c3c9\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:27.919908174Z\",\n      \"endTime\": \"2026-08-07T08:44:28.270170603Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"d8f3f6927a8ef77d465b40967c687f38513746bbb92b1f5ad929920293725d28","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"c3227703bc284fd311a90e89009e8ee622c07cad2c01a180da540a4dfc072b1a","test_name":"Customer should be able to create a HCP cluster and use cilium CNI plugin","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000c32740\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-lgwcbtllqrsl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e8ee892-8fb8-4c51-a421-0e8f90801ef5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e8ee892-8fb8-4c51-a421-0e8f90801ef5\",\n      \"name\": \"5e8ee892-8fb8-4c51-a421-0e8f90801ef5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:43.003779178Z\",\n      \"endTime\": \"2026-08-07T07:58:58.368511619Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster \"cilium-cl\" without CNI\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000c32740\u003e: \n    failed to create HCP cluster cilium-cl: failed waiting for cluster=\"cilium-cl\" in resourcegroup=\"cni-cilium-lgwcbtllqrsl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e8ee892-8fb8-4c51-a421-0e8f90801ef5\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5e8ee892-8fb8-4c51-a421-0e8f90801ef5\",\n      \"name\": \"5e8ee892-8fb8-4c51-a421-0e8f90801ef5\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:43.003779178Z\",\n      \"endTime\": \"2026-08-07T07:58:58.368511619Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"dbb08b128f21728275e681a87ed6265c7a25ce8422406325a5ceea3affe7367f","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"b2a22af13af1b85b6910ebb15dc000945d0c2831ac57e8cf138ded5f4284c7e2","test_name":"Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010105c0\u003e: \n    failed to create HCP cluster np-downgrade-78vv6q: failed waiting for cluster=\"np-downgrade-78vv6q\" in resourcegroup=\"rg-np-downgrade-78vv6q-8kzhzv86tmcz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/610194b0-0171-48a2-afe7-59baecacb1f0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/610194b0-0171-48a2-afe7-59baecacb1f0\",\n      \"name\": \"610194b0-0171-48a2-afe7-59baecacb1f0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:27.610486621Z\",\n      \"endTime\": \"2026-08-07T08:44:28.242585747Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0010105c0\u003e: \n    failed to create HCP cluster np-downgrade-78vv6q: failed waiting for cluster=\"np-downgrade-78vv6q\" in resourcegroup=\"rg-np-downgrade-78vv6q-8kzhzv86tmcz\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/610194b0-0171-48a2-afe7-59baecacb1f0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/610194b0-0171-48a2-afe7-59baecacb1f0\",\n      \"name\": \"610194b0-0171-48a2-afe7-59baecacb1f0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:25:27.610486621Z\",\n      \"endTime\": \"2026-08-07T08:44:28.242585747Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"dd0238a2cf95a469981cd0bad22ebc27231f602aa723d695808bcf6019ffc38f","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"a7944b9c5f72f76ef69cea7edebc5731988fa1ffb48cda97aa9e66edf4f90078","test_name":"Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-m2x787\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005af3e0\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-m2x787: failed waiting for cluster=\"cluster-zstream-4-20-m2x787\" in resourcegroup=\"rg-zstream-upgrade-4-20-p9sfgbm24ftk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c8fc4a8-92ae-4255-814d-e1362fd32f36\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c8fc4a8-92ae-4255-814d-e1362fd32f36\",\n      \"name\": \"5c8fc4a8-92ae-4255-814d-e1362fd32f36\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:57.357630105Z\",\n      \"endTime\": \"2026-08-07T08:21:59.441748911Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster \"cluster-zstream-4-20-m2x787\" with version 4.20.28 on candidate channel\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0005af3e0\u003e: \n    failed to create HCP cluster cluster-zstream-4-20-m2x787: failed waiting for cluster=\"cluster-zstream-4-20-m2x787\" in resourcegroup=\"rg-zstream-upgrade-4-20-p9sfgbm24ftk\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c8fc4a8-92ae-4255-814d-e1362fd32f36\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5c8fc4a8-92ae-4255-814d-e1362fd32f36\",\n      \"name\": \"5c8fc4a8-92ae-4255-814d-e1362fd32f36\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:57.357630105Z\",\n      \"endTime\": \"2026-08-07T08:21:59.441748911Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"e9bc3d6ae4c15d28d062170f2af1870ef3b620065fd8e0dffca1528c55f56565","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"eeecbe8da954785ad3bc9485fae6ca77c66153b617da9e1fa1a9114e437817b2","test_name":"Customer should be able to create an HCP cluster with back-level version 4.19","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:150]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc000e50180\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-br948jjgf9df\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-br948jjgf9df, error: context deadline exceeded\n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:150]: failed to create HCP cluster version 4.19\nUnexpected error:\n    \u003c*fmt.wrapErrors | 0xc000e50180\u003e: \n    failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-br948jjgf9df\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-br948jjgf9df, error: context deadline exceeded\n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","failure_pattern":"timeout during CreateHCPClusterAndWait; context deadline exceeded","search_query":"failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-br948jjgf9df\" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-br948jjgf9df, error: context deadline exceeded"}},{"row_id":"ec731d5fa41bf9babee5468c7e88ecc10374b1c313d4166b3e1b79cde1d20a7d","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"94b33bd0b5d3268c293f50324d6379cfc89f130ee4d70fc5143a51b8e729a2ca","test_name":"Customer should respect cluster-wide node limits with nodepool autoscaling","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0011829c0\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-25vh8cngktxn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5aeb61ff-6666-4e14-9f9e-018b08f7c733\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5aeb61ff-6666-4e14-9f9e-018b08f7c733\",\n      \"name\": \"5aeb61ff-6666-4e14-9f9e-018b08f7c733\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:42.604272672Z\",\n      \"endTime\": \"2026-08-07T08:23:48.961981626Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0011829c0\u003e: \n    failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-25vh8cngktxn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5aeb61ff-6666-4e14-9f9e-018b08f7c733\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5aeb61ff-6666-4e14-9f9e-018b08f7c733\",\n      \"name\": \"5aeb61ff-6666-4e14-9f9e-018b08f7c733\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:04:42.604272672Z\",\n      \"endTime\": \"2026-08-07T08:23:48.961981626Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"edfa8affb9b80b04c61773a9a037dae0079ab9da8712f6024039b1b99ee69217","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"9e5b66dcb2aefa897946026c7fd9b8a6fb826819a72fff75e17fd30ca9620253","test_name":"ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-5bnc2ggx8p9p/cluster-candidate-4-22-dq8mh5 should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e3c820\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-dq8mh5: failed waiting for cluster=\"cluster-candidate-4-22-dq8mh5\" in resourcegroup=\"rg-candidate-4-22-5bnc2ggx8p9p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b001b54a-7681-4219-9f1f-8468a7f95ee0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b001b54a-7681-4219-9f1f-8468a7f95ee0\",\n      \"name\": \"b001b54a-7681-4219-9f1f-8468a7f95ee0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:55.611664944Z\",\n      \"endTime\": \"2026-08-07T08:21:59.355617808Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-5bnc2ggx8p9p/cluster-candidate-4-22-dq8mh5 should provision\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000e3c820\u003e: \n    failed to create HCP cluster cluster-candidate-4-22-dq8mh5: failed waiting for cluster=\"cluster-candidate-4-22-dq8mh5\" in resourcegroup=\"rg-candidate-4-22-5bnc2ggx8p9p\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b001b54a-7681-4219-9f1f-8468a7f95ee0\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b001b54a-7681-4219-9f1f-8468a7f95ee0\",\n      \"name\": \"b001b54a-7681-4219-9f1f-8468a7f95ee0\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T08:02:55.611664944Z\",\n      \"endTime\": \"2026-08-07T08:21:59.355617808Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"fbb38f085e621d7cf2502d769a0d4d0ecd4e5bda0c79233ef59f8690304378fb","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"3a4a8898c39b14391c14badac4aaa14e6d75ddbfca4c33d01a44febee1d5b958","test_name":"Customer should not be able to reuse subnets and NSGs between clusters","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006a58c0\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-j9x4nssnf48t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/600512e3-57bf-44d4-b27f-c782dc21fb12\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/600512e3-57bf-44d4-b27f-c782dc21fb12\",\n      \"name\": \"600512e3-57bf-44d4-b27f-c782dc21fb12\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:44.6384065Z\",\n      \"endTime\": \"2026-08-07T07:58:48.395856054Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0006a58c0\u003e: \n    failed to create HCP cluster basic-cluster: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-j9x4nssnf48t\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/600512e3-57bf-44d4-b27f-c782dc21fb12\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/600512e3-57bf-44d4-b27f-c782dc21fb12\",\n      \"name\": \"600512e3-57bf-44d4-b27f-c782dc21fb12\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:44.6384065Z\",\n      \"endTime\": \"2026-08-07T07:58:48.395856054Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}},{"row_id":"fcc326c11905c57bbbe5e1922fa3af37e4f158bbcba711abcd80923f4dbf4dab","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"d712d5ce17264f0ef550496665e2a7bd7d67dfac479ff00deed5bcb5e610f7c9","test_name":"ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.003s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.003s.\ncluster should eventually succeed after role assignments are created\nThe function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:\ncluster entered terminal Failed state after role assignment deployment\nExpected\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...\nnot to equal\n    \u003carmredhatopenshifthcp.ProvisioningState\u003e: ...","failure_pattern_match":{"status":"clustered","failure_pattern_id":"7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","failure_pattern":"cluster entered terminal Failed state after role assignment deployment","search_query":"cluster entered terminal Failed state after role assignment deployment"}},{"row_id":"fe5396a9df59721b33a52768e784eee8f4833e2815e0e425fa5d504810e3655e","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085627580924301312","occurred_at":"2026-08-07T07:22:18Z","failed_at":"e2e","signature_id":"bd5c059b837307cf24f537d32d322eefef0077e92d58c789ecce557e6ce0da03","test_name":"Customer should be able to create an HCP cluster then delete it by deleting the customer resource group","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009d69c0\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-tlqbd2j6hf25\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ef9852f-682d-495a-99a0-fd24fdd67370\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ef9852f-682d-495a-99a0-fd24fdd67370\",\n      \"name\": \"7ef9852f-682d-495a-99a0-fd24fdd67370\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:44.262783411Z\",\n      \"endTime\": \"2026-08-07T07:58:48.329050999Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster \"cx-rg-hcp-cluster\"\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc0009d69c0\u003e: \n    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster=\"cx-rg-hcp-cluster\" in resourcegroup=\"cx-rg-cluster-tlqbd2j6hf25\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ef9852f-682d-495a-99a0-fd24fdd67370\n    --------------------------------------------------------------------------------\n    RESPONSE 200: 200 OK\n    ERROR CODE: InternalServerError\n    --------------------------------------------------------------------------------\n    {\n      \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ef9852f-682d-495a-99a0-fd24fdd67370\",\n      \"name\": \"7ef9852f-682d-495a-99a0-fd24fdd67370\",\n      \"status\": \"Failed\",\n      \"startTime\": \"2026-08-07T07:39:44.262783411Z\",\n      \"endTime\": \"2026-08-07T07:58:48.329050999Z\",\n      \"error\": {\n        \"code\": \"InternalServerError\",\n        \"message\": \"[clusterServiceClusterStatus] \\u003cno_message\\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","failure_pattern":"ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: InternalServerError"}}],"failure_pattern_summary":{"signature_count":45,"failure_pattern_ids":["0ece011306dab74cd22dd19f1f3c89bbdb930a2321b61be59e4a6a90aae6e3eb","7207c83a34a191366f94a6a113fc9a5aaabe50fb5e6befcd21b6e21322f73a87","ac227089170e13ef6a47a3b4633ad2059cb4ac90e0b8628a57dcc930f8b4a4a2","ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f","e9065928ca1d323095a060fb425d3a601fe5817c1279001c85a6c30b326c861b"],"matched_occurrences":45,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}},{"run":{"environment":"int","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085552497933422592","job_name":"branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel","pr_number":0,"pr_state":"","pr_sha":"","final_merged_sha":"","merged_pr":true,"post_good_commit":true,"failed":true,"occurred_at":"2026-08-07T02:23:57Z"},"failed_at":["e2e"],"failed_test_count":2,"occurrences":[{"row_id":"a853d04b6676d88faece5326b63be53410e41cf67f0e622ec24e3342a685ab91","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085552497933422592","occurred_at":"2026-08-07T02:23:57Z","failed_at":"e2e","signature_id":"7269898a1029a630ccb580ccffd8c4ae1c6eb7659de2a27a8754827626d9330c","test_name":"Customer should create a cluster with private ingress using v20260630preview and verify the ingress is internal","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:91]: v20260630preview API still not deployed as of 2026-08-07T00:00:00Z deadline\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:91]: v20260630preview API still not deployed as of 2026-08-07T00:00:00Z deadline","failure_pattern_match":{"status":"clustered","failure_pattern_id":"94cd26658412afd2794ba5c332d1af73e780853eed8982883da6e3a0e15f5951","failure_pattern":"v20260630preview API still not deployed as of \u003ctimestamp\u003e deadline","search_query":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:91]: v20260630preview API still not deployed as of 2026-08-07T00:00:00Z deadline"}},{"row_id":"b6331d9e8f70da31b385a8b7d655452c855708209da7557e41eba52453de0775","run_url":"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel/2085552497933422592","occurred_at":"2026-08-07T02:23:57Z","failed_at":"e2e","signature_id":"d75836a17aca7a0c89965bae888e881618593b0fcd436782b91afaa479ff76b0","test_name":"FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property","test_suite":"integration/parallel","failure_text":"fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ee4bc0\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-b9rxznrnjjbc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f311d32b-4512-4996-ba90-b9a0b80a3f06\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred\nfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:120]: failed to create HCP cluster \"fips-enabled-cluster\" with cryptoRestrictions set to FIPS\nUnexpected error:\n    \u003c*fmt.wrapError | 0xc000ee4bc0\u003e: \n    failed waiting for cluster=\"fips-enabled-cluster\" in resourcegroup=\"fips-enabled-b9rxznrnjjbc\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f311d32b-4512-4996-ba90-b9a0b80a3f06\n    --------------------------------------------------------------------------------\n    RESPONSE 400: 400 Bad Request\n    ERROR CODE: NoRegisteredProviderFound\n    --------------------------------------------------------------------------------\n    {\n      \"error\": {\n        \"code\": \"NoRegisteredProviderFound\",\n        \"message\": \"No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.\"\n      }\n    }\n    --------------------------------------------------------------------------------\n    \n    ...\noccurred","failure_pattern_match":{"status":"clustered","failure_pattern_id":"ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f","failure_pattern":"ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'uksouth' and API version '2026-06-30-preview' for type 'locations/hcpOperationStatuses'. The supported api-versions are '2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth'.; provider Microsoft.RedHatOpenShift","search_query":"ERROR CODE: NoRegisteredProviderFound"}}],"failure_pattern_summary":{"signature_count":2,"failure_pattern_ids":["94cd26658412afd2794ba5c332d1af73e780853eed8982883da6e3a0e15f5951","ba4588eaec0a1624f3e9e2b68b856f9c28fafa454d0205c94f35610f5416c93f"],"matched_occurrences":2,"unmatched_occurrences":0,"match_summary":"multiple_clustered"}}]}]}