CIHealth Run Log

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:46:15	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
23:42:00	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	33	e2e (32) timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000f54d20>: failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-w2ndls, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-w2ndls" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-w2ndls, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000f54d20>: failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-w2ndls, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-w2ndls" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-w2ndls, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapErrors \| 0xc001736fc0>: failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qmrn9q, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qmrn9q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qmrn9q, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapErrors \| 0xc001736fc0>: failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qmrn9q, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qmrn9q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-qmrn9q, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc0003805d0>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gstdrp, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-gstdrp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gstdrp, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc0003805d0>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gstdrp, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-gstdrp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gstdrp, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: Unexpected error: <fmt.wrapErrors \| 0xc000b62f90>: failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-mvd2qk, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-mvd2qk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-mvd2qk, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: Unexpected error: <fmt.wrapErrors \| 0xc000b62f90>: failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-mvd2qk, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-mvd2qk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-mvd2qk, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775785195 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to use workload identity via the cluster OIDC issuer URL [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169 "ts"="2026-04-10 01:39:55.131080" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-use-workload-identity-via-the-cluster-OIDC-issuer-URL\|pid:903" "ts"="2026-04-10 01:39:55.137582" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 01:39:55.137 "ts"="2026-04-10 01:39:55.137651" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-dnbtp6" STEP: creating cluster parameters @ 04/10/26 01:39:59.261 STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/10/26 01:39:59.261 STEP: creating the HCP cluster @ 04/10/26 01:41:09.86 "ts"="2026-04-10 01:41:09.861125" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-dnbtp6" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775785195 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to use workload identity via the cluster OIDC issuer URL [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169 "ts"="2026-04-10 01:39:55.131080" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-use-workload-identity-via-the-cluster-OIDC-issuer-URL\|pid:903" "ts"="2026-04-10 01:39:55.137582" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 01:39:55.137 "ts"="2026-04-10 01:39:55.137651" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-dnbtp6" STEP: creating cluster parameters @ 04/10/26 01:39:59.261 STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/10/26 01:39:59.261 STEP: creating the HCP cluster @ 04/10/26 01:41:09.86 "ts"="2026-04-10 01:41:09.861125" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-dnbtp6" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapErrors \| 0xc0012a5590>: failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-49vb8b, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-49vb8b" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-49vb8b, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapErrors \| 0xc0012a5590>: failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-49vb8b, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-49vb8b" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-49vb8b, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc0010fa720>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-tk5vh5, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-tk5vh5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-tk5vh5, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc0010fa720>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-tk5vh5, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-tk5vh5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-tk5vh5, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc0008c6930>: failed to create HCP cluster cp-ystream-upgrade-4-21-8flt69, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-8flt69 in resource group rg-cp-ystream-upgrade-4-21-xqc5mj, error: failed waiting for cluster="cp-ystream-upgrade-4-21-8flt69" in resourcegroup="rg-cp-ystream-upgrade-4-21-xqc5mj" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-8flt69 in resource group rg-cp-ystream-upgrade-4-21-xqc5mj, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc0008c6930>: failed to create HCP cluster cp-ystream-upgrade-4-21-8flt69, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-8flt69 in resource group rg-cp-ystream-upgrade-4-21-xqc5mj, error: failed waiting for cluster="cp-ystream-upgrade-4-21-8flt69" in resourcegroup="rg-cp-ystream-upgrade-4-21-xqc5mj" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-8flt69 in resource group rg-cp-ystream-upgrade-4-21-xqc5mj, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc0011f2e10>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-48kzhd, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-48kzhd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-48kzhd, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc0011f2e10>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-48kzhd, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-48kzhd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-48kzhd, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.995667" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.997695" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.998987" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:26.000957" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:26.003463" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:26.005539" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:26.007826" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:26.010162" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.012551" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.014656" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.016406" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.376864" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=118 "retryIn"="1m0s" "elapsed"="1h57m2s" "ts"="2026-04-10 01:43:28.576766" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=119 "elapsed"="1h58m3s" STEP: creating resource group for admin credential lifecycle testing @ 04/10/26 01:43:28.576 "ts"="2026-04-10 01:43:28.576865" "level"=0 "msg"="creating resource group" "resourceGroup"="admin-credential-lifecycle-test-r46lz5" STEP: creating cluster parameters @ 04/10/26 01:43:32.738 STEP: creating customer resources @ 04/10/26 01:43:32.738 STEP: starting HCP cluster creation asynchronously @ 04/10/26 01:44:53.672 "ts"="2026-04-10 01:44:53.672604" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="admin-cred-lifecycle-85ks84" "resourceGroup"="admin-credential-lifecycle-test-r46lz5" STEP: waiting for cluster to appear and testing admin credentials while in deploying state @ 04/10/26 01:44:54.783 "ts"="2026-04-10 01:44:54.784022" "level"=0 "msg"="creating cluster, waiting for it to reach a terminal state" "ts"="2026-04-10 01:44:55.173149" "level"=0 "msg"="Cluster provisioning state updated" "provisioningState"="Accepted" STEP: testing admin credentials while cluster is in deploying state @ 04/10/26 01:44:55.173 STEP: verifying admin credentials request fails with HTTP 409 CONFLICT on deploying cluster @ 04/10/26 01:44:55.772 "ts"="2026-04-10 01:44:55.772589" "level"=0 "msg"="Admin credentials request correctly returned 409 conflict error while cluster is deploying" time="2026-04-10T01:45:22Z" level=info msg="Received signal." signal=interrupt Interrupted, terminating testsCommand Error: signal: killed Deserializaion Error: no output from command free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.995667" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.997695" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.998987" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:26.000957" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:26.003463" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:26.005539" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:26.007826" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:26.010162" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.012551" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.014656" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.016406" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.376864" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=118 "retryIn"="1m0s" "elapsed"="1h57m2s" "ts"="2026-04-10 01:43:28.576766" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=119 "elapsed"="1h58m3s" STEP: creating resource group for admin credential lifecycle testing @ 04/10/26 01:43:28.576 "ts"="2026-04-10 01:43:28.576865" "level"=0 "msg"="creating resource group" "resourceGroup"="admin-credential-lifecycle-test-r46lz5" STEP: creating cluster parameters @ 04/10/26 01:43:32.738 STEP: creating customer resources @ 04/10/26 01:43:32.738 STEP: starting HCP cluster creation asynchronously @ 04/10/26 01:44:53.672 "ts"="2026-04-10 01:44:53.672604" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="admin-cred-lifecycle-85ks84" "resourceGroup"="admin-credential-lifecycle-test-r46lz5" STEP: waiting for cluster to appear and testing admin credentials while in deploying state @ 04/10/26 01:44:54.783 "ts"="2026-04-10 01:44:54.784022" "level"=0 "msg"="creating cluster, waiting for it to reach a terminal state" "ts"="2026-04-10 01:44:55.173149" "level"=0 "msg"="Cluster provisioning state updated" "provisioningState"="Accepted" STEP: testing admin credentials while cluster is in deploying state @ 04/10/26 01:44:55.173 STEP: verifying admin credentials request fails with HTTP 409 CONFLICT on deploying cluster @ 04/10/26 01:44:55.772 "ts"="2026-04-10 01:44:55.772589" "level"=0 "msg"="Admin credentials request correctly returned 409 conflict error while cluster is deploying" time="2026-04-10T01:45:22Z" level=info msg="Received signal." signal=interrupt Interrupted, terminating testsCommand Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc001887080>: failed to create HCP cluster cluster-zstream-4-19-pjnf9v, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pjnf9v in resource group rg-zstream-upgrade-4-19-vwgn9q, error: failed waiting for cluster="cluster-zstream-4-19-pjnf9v" in resourcegroup="rg-zstream-upgrade-4-19-vwgn9q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pjnf9v in resource group rg-zstream-upgrade-4-19-vwgn9q, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc001887080>: failed to create HCP cluster cluster-zstream-4-19-pjnf9v, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pjnf9v in resource group rg-zstream-upgrade-4-19-vwgn9q, error: failed waiting for cluster="cluster-zstream-4-19-pjnf9v" in resourcegroup="rg-zstream-upgrade-4-19-vwgn9q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-pjnf9v in resource group rg-zstream-upgrade-4-19-vwgn9q, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775785253 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 01:40:53.520546" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:917" "ts"="2026-04-10 01:40:53.527501" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 01:40:53.527 "ts"="2026-04-10 01:40:53.527553" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-fj8dkb" STEP: creating cluster parameters @ 04/10/26 01:40:57.823 STEP: creating customer resources @ 04/10/26 01:40:57.823 STEP: creating the HCP cluster @ 04/10/26 01:42:18.762 "ts"="2026-04-10 01:42:18.762988" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-fj8dkb" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775785253 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 01:40:53.520546" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:917" "ts"="2026-04-10 01:40:53.527501" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 01:40:53.527 "ts"="2026-04-10 01:40:53.527553" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-fj8dkb" STEP: creating cluster parameters @ 04/10/26 01:40:57.823 STEP: creating customer resources @ 04/10/26 01:40:57.823 STEP: creating the HCP cluster @ 04/10/26 01:42:18.762 "ts"="2026-04-10 01:42:18.762988" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-fj8dkb" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc0009527b0>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5fkhxr, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5fkhxr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5fkhxr, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc0009527b0>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5fkhxr, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5fkhxr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5fkhxr, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure signed" "attempt"=101 "retryIn"="1m0s" "elapsed"="1h40m0s" "ts"="2026-04-10 01:26:25.970126" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.972003" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.974055" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.976115" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:25.978443" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.980652" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.983036" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.985005" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:25.986981" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:25.988915" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:25.990940" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:25.992341" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:25.994468" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:25.995916" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:25.999219" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.001652" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.362711" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=118 "elapsed"="1h57m2s" STEP: creating a resource group @ 04/10/26 01:42:27.362 "ts"="2026-04-10 01:42:27.362860" "level"=0 "msg"="creating resource group" "resourceGroup"="tls-endpoint-cluster-k5wl7j" STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/10/26 01:42:31.242 STEP: creating a standard hcp cluster @ 04/10/26 01:43:33.26 "ts"="2026-04-10 01:43:33.260551" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="tls-endpoint-hcp-cluster" "resourceGroup"="tls-endpoint-cluster-k5wl7j" Command Error: signal: killed Deserializaion Error: no output from command signed" "attempt"=101 "retryIn"="1m0s" "elapsed"="1h40m0s" "ts"="2026-04-10 01:26:25.970126" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.972003" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.974055" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.976115" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:25.978443" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.980652" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.983036" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.985005" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:25.986981" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:25.988915" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:25.990940" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:25.992341" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:25.994468" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:25.995916" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:25.999219" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.001652" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.362711" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=118 "elapsed"="1h57m2s" STEP: creating a resource group @ 04/10/26 01:42:27.362 "ts"="2026-04-10 01:42:27.362860" "level"=0 "msg"="creating resource group" "resourceGroup"="tls-endpoint-cluster-k5wl7j" STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/10/26 01:42:31.242 STEP: creating a standard hcp cluster @ 04/10/26 01:43:33.26 "ts"="2026-04-10 01:43:33.260551" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="tls-endpoint-hcp-cluster" "resourceGroup"="tls-endpoint-cluster-k5wl7j" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc0008f6090>: failed to create HCP cluster cluster-zstream-4-20-27sf7p, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-27sf7p in resource group rg-zstream-upgrade-4-20-jh4tp7, error: failed waiting for cluster="cluster-zstream-4-20-27sf7p" in resourcegroup="rg-zstream-upgrade-4-20-jh4tp7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-27sf7p in resource group rg-zstream-upgrade-4-20-jh4tp7, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc0008f6090>: failed to create HCP cluster cluster-zstream-4-20-27sf7p, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-27sf7p in resource group rg-zstream-upgrade-4-20-jh4tp7, error: failed waiting for cluster="cluster-zstream-4-20-27sf7p" in resourcegroup="rg-zstream-upgrade-4-20-jh4tp7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-27sf7p in resource group rg-zstream-upgrade-4-20-jh4tp7, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapErrors \| 0xc0012da690>: failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-r8p9n9, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-r8p9n9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-r8p9n9, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapErrors \| 0xc0012da690>: failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-r8p9n9, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-r8p9n9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-r8p9n9, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapErrors \| 0xc000960ea0>: failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-tsvtqx, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-tsvtqx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-tsvtqx, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapErrors \| 0xc000960ea0>: failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-tsvtqx, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-tsvtqx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-tsvtqx, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775785307 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41 "ts"="2026-04-10 01:41:47.102900" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-not-be-able-to-perform-various-invalid-operations-on-cluster-resources\|pid:930" "ts"="2026-04-10 01:41:49.288805" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="2s" STEP: creating a resource group @ 04/10/26 01:41:49.288 "ts"="2026-04-10 01:41:49.288894" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-6b9z9j" STEP: creating cluster parameters @ 04/10/26 01:41:54.08 STEP: creating customer resources @ 04/10/26 01:41:54.081 STEP: creating the HCP cluster @ 04/10/26 01:43:05.737 "ts"="2026-04-10 01:43:05.737229" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-6b9z9j" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775785307 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41 "ts"="2026-04-10 01:41:47.102900" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-not-be-able-to-perform-various-invalid-operations-on-cluster-resources\|pid:930" "ts"="2026-04-10 01:41:49.288805" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="2s" STEP: creating a resource group @ 04/10/26 01:41:49.288 "ts"="2026-04-10 01:41:49.288894" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-6b9z9j" STEP: creating cluster parameters @ 04/10/26 01:41:54.08 STEP: creating customer resources @ 04/10/26 01:41:54.081 STEP: creating the HCP cluster @ 04/10/26 01:43:05.737 "ts"="2026-04-10 01:43:05.737229" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-6b9z9j" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000f6ecc0>: failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wdgc8m, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-wdgc8m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wdgc8m, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000f6ecc0>: failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wdgc8m, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-wdgc8m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-wdgc8m, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:94]: Unexpected error: <fmt.wrapErrors \| 0xc001487230>: failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-pz5zl4, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pz5zl4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-pz5zl4, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:94]: Unexpected error: <fmt.wrapErrors \| 0xc001487230>: failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-pz5zl4, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pz5zl4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-pz5zl4, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc00180eba0>: failed to create HCP cluster cluster-zstream-4-21-wlj52v, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-wlj52v in resource group rg-zstream-upgrade-4-21-pzzmrb, error: failed waiting for cluster="cluster-zstream-4-21-wlj52v" in resourcegroup="rg-zstream-upgrade-4-21-pzzmrb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-wlj52v in resource group rg-zstream-upgrade-4-21-pzzmrb, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc00180eba0>: failed to create HCP cluster cluster-zstream-4-21-wlj52v, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-wlj52v in resource group rg-zstream-upgrade-4-21-pzzmrb, error: failed waiting for cluster="cluster-zstream-4-21-wlj52v" in resourcegroup="rg-zstream-upgrade-4-21-pzzmrb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-wlj52v in resource group rg-zstream-upgrade-4-21-pzzmrb, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775778325 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group [PreLaunchSetup:None, ARO-HCP-RP-API-Compatible, Importance:Critical, Positivity:Positive] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35 "ts"="2026-04-09 23:45:25.749409" "level"=0 "msg"="Starting identity container acquisition" "count"=3 "specID"="Customer-should-be-able-to-create-several-HCP-clusters-in-their-customer-resource-group,-but-not-in-the-same-managed-resource-group\|pid:120" "ts"="2026-04-09 23:45:25.773221" "level"=0 "msg"="Successfully assigned identity containers" "count"=3 "attempt"=1 "elapsed"="0s" STEP: creating a shared customer resource group @ 04/09/26 23:45:25.773 "ts"="2026-04-09 23:45:25.773310" "level"=0 "msg"="creating resource group" "resourceGroup"="customer-rg-xmtqnl" STEP: creating customer infrastructure and managed identities for first cluster @ 04/09/26 23:45:29.579 STEP: creating customer infrastructure and managed identities for second cluster @ 04/09/26 23:46:53.764 STEP: starting creation of both clusters in parallel @ 04/09/26 23:48:04.669 "ts"="2026-04-09 23:48:04.669788" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster" "resourceGroup"="customer-rg-xmtqnl" "ts"="2026-04-09 23:48:13.377207" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster2" "resourceGroup"="customer-rg-xmtqnl" STEP: waiting for first cluster to complete creation @ 04/09/26 23:48:26.383 Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775778325 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group [PreLaunchSetup:None, ARO-HCP-RP-API-Compatible, Importance:Critical, Positivity:Positive] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35 "ts"="2026-04-09 23:45:25.749409" "level"=0 "msg"="Starting identity container acquisition" "count"=3 "specID"="Customer-should-be-able-to-create-several-HCP-clusters-in-their-customer-resource-group,-but-not-in-the-same-managed-resource-group\|pid:120" "ts"="2026-04-09 23:45:25.773221" "level"=0 "msg"="Successfully assigned identity containers" "count"=3 "attempt"=1 "elapsed"="0s" STEP: creating a shared customer resource group @ 04/09/26 23:45:25.773 "ts"="2026-04-09 23:45:25.773310" "level"=0 "msg"="creating resource group" "resourceGroup"="customer-rg-xmtqnl" STEP: creating customer infrastructure and managed identities for first cluster @ 04/09/26 23:45:29.579 STEP: creating customer infrastructure and managed identities for second cluster @ 04/09/26 23:46:53.764 STEP: starting creation of both clusters in parallel @ 04/09/26 23:48:04.669 "ts"="2026-04-09 23:48:04.669788" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster" "resourceGroup"="customer-rg-xmtqnl" "ts"="2026-04-09 23:48:13.377207" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster2" "resourceGroup"="customer-rg-xmtqnl" STEP: waiting for first cluster to complete creation @ 04/09/26 23:48:26.383 Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure ners but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.994626" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.996713" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.999752" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:26.001403" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:26.003322" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:26.005122" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:26.007258" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:26.009086" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:26.011684" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:26.013517" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:26.015514" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:26.017919" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.020182" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.022344" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.024553" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.378309" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=118 "retryIn"="1m0s" "elapsed"="1h57m2s" "ts"="2026-04-10 01:43:28.584897" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=119 "elapsed"="1h58m3s" STEP: creating a resource group @ 04/10/26 01:43:28.584 "ts"="2026-04-10 01:43:28.584987" "level"=0 "msg"="creating resource group" "resourceGroup"="patch-tags-zgdlx7" STEP: creating cluster parameters @ 04/10/26 01:43:32.587 STEP: creating customer resources @ 04/10/26 01:43:32.587 STEP: creating the HCP cluster @ 04/10/26 01:44:53.237 "ts"="2026-04-10 01:44:53.238008" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="patch-tags-cluster" "resourceGroup"="patch-tags-zgdlx7" Command Error: signal: killed Deserializaion Error: no output from command ners but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.994626" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.996713" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.999752" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:26.001403" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:26.003322" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:26.005122" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:26.007258" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:26.009086" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:26.011684" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:26.013517" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:26.015514" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:26.017919" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.020182" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.022344" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.024553" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.378309" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=118 "retryIn"="1m0s" "elapsed"="1h57m2s" "ts"="2026-04-10 01:43:28.584897" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=119 "elapsed"="1h58m3s" STEP: creating a resource group @ 04/10/26 01:43:28.584 "ts"="2026-04-10 01:43:28.584987" "level"=0 "msg"="creating resource group" "resourceGroup"="patch-tags-zgdlx7" STEP: creating cluster parameters @ 04/10/26 01:43:32.587 STEP: creating customer resources @ 04/10/26 01:43:32.587 STEP: creating the HCP cluster @ 04/10/26 01:44:53.237 "ts"="2026-04-10 01:44:53.238008" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="patch-tags-cluster" "resourceGroup"="patch-tags-zgdlx7" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure ity containers: requested 1 identity containers but only 0 are assigned" "attempt"=101 "retryIn"="1m0s" "elapsed"="1h40m0s" "ts"="2026-04-10 01:26:25.976307" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.978331" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.979738" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.982005" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:25.984519" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.986359" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.988639" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.990629" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:25.993262" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:25.995084" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:25.997308" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:25.999692" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:26.001244" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.003164" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.004509" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.006500" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.375412" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=118 "elapsed"="1h57m2s" STEP: creating a resource group @ 04/10/26 01:42:27.375 "ts"="2026-04-10 01:42:27.375486" "level"=0 "msg"="creating resource group" "resourceGroup"="autoscaling-cluster-vtfl2j" STEP: creating customer resources @ 04/10/26 01:42:31.654 STEP: creating the cluster @ 04/10/26 01:43:40.574 "ts"="2026-04-10 01:43:40.574885" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="autoscaling-hcp-cluster" "resourceGroup"="autoscaling-cluster-vtfl2j" Command Error: signal: killed Deserializaion Error: no output from command ity containers: requested 1 identity containers but only 0 are assigned" "attempt"=101 "retryIn"="1m0s" "elapsed"="1h40m0s" "ts"="2026-04-10 01:26:25.976307" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.978331" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.979738" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.982005" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:25.984519" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.986359" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.988639" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.990629" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:25.993262" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:25.995084" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:25.997308" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:25.999692" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:26.001244" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.003164" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.004509" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.006500" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.375412" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=118 "elapsed"="1h57m2s" STEP: creating a resource group @ 04/10/26 01:42:27.375 "ts"="2026-04-10 01:42:27.375486" "level"=0 "msg"="creating resource group" "resourceGroup"="autoscaling-cluster-vtfl2j" STEP: creating customer resources @ 04/10/26 01:42:31.654 STEP: creating the cluster @ 04/10/26 01:43:40.574 "ts"="2026-04-10 01:43:40.574885" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="autoscaling-hcp-cluster" "resourceGroup"="autoscaling-cluster-vtfl2j" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error: <fmt.wrapErrors \| 0xc001345860>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-7hbmqr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-7hbmqr, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error: <fmt.wrapErrors \| 0xc001345860>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-7hbmqr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-7hbmqr, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapErrors \| 0xc0011f0ed0>: failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-gnrpbw, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gnrpbw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-gnrpbw, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapErrors \| 0xc0011f0ed0>: failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-gnrpbw, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gnrpbw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-gnrpbw, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc000ff67b0>: failed to create HCP cluster np-version-upgrade-cluster-77fx42, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-77fx42 in resource group rg-np-version-upgrade-77fx42-prfhcc, error: failed waiting for cluster="np-version-upgrade-cluster-77fx42" in resourcegroup="rg-np-version-upgrade-77fx42-prfhcc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-77fx42 in resource group rg-np-version-upgrade-77fx42-prfhcc, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc000ff67b0>: failed to create HCP cluster np-version-upgrade-cluster-77fx42, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-77fx42 in resource group rg-np-version-upgrade-77fx42-prfhcc, error: failed waiting for cluster="np-version-upgrade-cluster-77fx42" in resourcegroup="rg-np-version-upgrade-77fx42-prfhcc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-77fx42 in resource group rg-np-version-upgrade-77fx42-prfhcc, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapErrors \| 0xc000e950e0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6gfpn9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6gfpn9, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapErrors \| 0xc000e950e0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-6gfpn9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-6gfpn9, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775782039 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 00:47:19.461902 876 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 00:47:19.544986 876 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 00:47:19.545244" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:876" "ts"="2026-04-10 00:47:19.551002" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 00:47:19.551 "ts"="2026-04-10 00:47:19.551072" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" STEP: creating cluster parameters at control plane version @ 04/10/26 00:47:23.522 W0410 00:47:23.605176 876 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 00:47:23.605 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 00:48:44.534 "ts"="2026-04-10 00:48:44.534430" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-9s68vn" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:129 @ 04/10/26 01:33:44.539 "ts"="2026-04-10 01:33:44.540116" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 01:33:44.540243" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-9s68vn-vkqvp9" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "ts"="2026-04-10 01:33:44.866362" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-9s68vn-vkqvp9" "resourceGroup"="" "ts"="2026-04-10 01:33:47.540364" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 01:33:47.540506" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 01:33:47.540574" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 01:33:47.541736" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "ts"="2026-04-10 01:33:47.541753" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "ts"="2026-04-10 01:33:47.921967" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "deployment"="pooledNonMsiScopedAssignments-gs3eml53zo4am" "ts"="2026-04-10 01:33:48.225104" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "deployment"="customer-infra-np-version-upgrade-cluster-9s68vn-vkqvp9" "ts"="2026-04-10 01:33:48.267881" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 01:33:48.835002" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="msi-deployment-148" "ts"="2026-04-10 01:33:48.983763" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 01:33:48.983864" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 01:33:48.983978" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775782039 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 00:47:19.461902 876 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 00:47:19.544986 876 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 00:47:19.545244" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:876" "ts"="2026-04-10 00:47:19.551002" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 00:47:19.551 "ts"="2026-04-10 00:47:19.551072" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" STEP: creating cluster parameters at control plane version @ 04/10/26 00:47:23.522 W0410 00:47:23.605176 876 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 00:47:23.605 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 00:48:44.534 "ts"="2026-04-10 00:48:44.534430" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-9s68vn" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:129 @ 04/10/26 01:33:44.539 "ts"="2026-04-10 01:33:44.540116" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 01:33:44.540243" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-9s68vn-vkqvp9" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "ts"="2026-04-10 01:33:44.866362" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-9s68vn-vkqvp9" "resourceGroup"="" "ts"="2026-04-10 01:33:47.540364" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 01:33:47.540506" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 01:33:47.540574" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 01:33:47.541736" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "ts"="2026-04-10 01:33:47.541753" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "ts"="2026-04-10 01:33:47.921967" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "deployment"="pooledNonMsiScopedAssignments-gs3eml53zo4am" "ts"="2026-04-10 01:33:48.225104" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" "deployment"="customer-infra-np-version-upgrade-cluster-9s68vn-vkqvp9" "ts"="2026-04-10 01:33:48.267881" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 01:33:48.835002" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="msi-deployment-148" "ts"="2026-04-10 01:33:48.983763" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 01:33:48.983864" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 01:33:48.983978" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-9s68vn-v74ng7" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:116]: Unexpected error: <fmt.wrapErrors \| 0xc0009a15f0>: failed waiting for cluster="cilium-cluster" in resourcegroup="e2e-cilium-5h55f6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-5h55f6, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:116]: Unexpected error: <fmt.wrapErrors \| 0xc0009a15f0>: failed waiting for cluster="cilium-cluster" in resourcegroup="e2e-cilium-5h55f6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-5h55f6, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 23:42:00 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite prod/parallel Show raw failure ity containers but only 0 are assigned" "attempt"=101 "retryIn"="1m0s" "elapsed"="1h40m0s" "ts"="2026-04-10 01:26:25.973178" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.975182" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.977194" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.979493" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:25.981426" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.983561" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.985249" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.987325" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:25.989229" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:25.991218" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:25.993463" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:25.995876" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:25.997941" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.000365" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.002150" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.004219" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.369138" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=118 "elapsed"="1h57m2s" STEP: creating resource group @ 04/10/26 01:42:27.369 "ts"="2026-04-10 01:42:27.369233" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-stable-4-20-c5n4s2" STEP: creating customer resources @ 04/10/26 01:42:31.433 STEP: creating the HCP cluster with version '4.20' on stable channel @ 04/10/26 01:43:32.288 "ts"="2026-04-10 01:43:32.288564" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-stable-4-20-4vbjtk" "resourceGroup"="rg-stable-4-20-c5n4s2" Command Error: signal: killed Deserializaion Error: no output from command ity containers but only 0 are assigned" "attempt"=101 "retryIn"="1m0s" "elapsed"="1h40m0s" "ts"="2026-04-10 01:26:25.973178" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 01:27:25.975182" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 01:28:25.977194" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 01:29:25.979493" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 01:30:25.981426" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 01:31:25.983561" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 01:32:25.985249" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 01:33:25.987325" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 01:34:25.989229" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 01:35:25.991218" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 01:36:25.993463" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 01:37:25.995876" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 01:38:25.997941" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 01:39:26.000365" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 01:40:26.002150" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 01:41:26.004219" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 01:42:27.369138" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=118 "elapsed"="1h57m2s" STEP: creating resource group @ 04/10/26 01:42:27.369 "ts"="2026-04-10 01:42:27.369233" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-stable-4-20-c5n4s2" STEP: creating customer resources @ 04/10/26 01:42:31.433 STEP: creating the HCP cluster with version '4.20' on stable channel @ 04/10/26 01:43:32.288 "ts"="2026-04-10 01:43:32.288564" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-stable-4-20-4vbjtk" "resourceGroup"="rg-stable-4-20-c5n4s2" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 23:42:00 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc000906510>: failed to create HCP cluster ea-list7qgk62, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-list7qgk62 in resource group ea-list-dsv98b, error: failed waiting for cluster="ea-list7qgk62" in resourcegroup="ea-list-dsv98b" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-list7qgk62 in resource group ea-list-dsv98b, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc000906510>: failed to create HCP cluster ea-list7qgk62, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-list7qgk62 in resource group ea-list-dsv98b, error: failed waiting for cluster="ea-list7qgk62" in resourcegroup="ea-list-dsv98b" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-list7qgk62 in resource group ea-list-dsv98b, error: context deadline exceeded ... occurred unknown (1) Process did not finish before 2h0m0s timeout 23:42:00 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T01:45:22Z"}
23:20:05	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	unknown	failed	n/a	1	non-artifact-backed failure (no junit artifacts)
22:45:00	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
22:41:54	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
19:47:11	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
10:43:15	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	7	e2e (7) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 10:43:15 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:165]: Unexpected error: <fmt.wrapError \| 0xc000a4e0c0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-vpz2m5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:165]: Unexpected error: <fmt.wrapError \| 0xc000a4e0c0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-vpz2m5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 10:43:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error: <fmt.wrapError \| 0xc0008d92c0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-ddwscf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error: <fmt.wrapError \| 0xc0008d92c0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-ddwscf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred Interrupted by User 10:43:15 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift 10:43:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error: <exported.ResponseError \| 0xc001404990>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rclvn7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error: <exported.ResponseError \| 0xc001404990>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rclvn7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 10:43:15 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapError \| 0xc0016011a0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2668zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-wkpcsh/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapError \| 0xc0016011a0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-2668zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-wkpcsh/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 10:43:15 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error: <fmt.wrapError \| 0xc001418660>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-k2bkdp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error: <fmt.wrapError \| 0xc001418660>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-k2bkdp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 10:43:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error: <fmt.wrapError \| 0xc00141aa20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5jk444/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error: <fmt.wrapError \| 0xc00141aa20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5jk444/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred
02:13:13	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	10	e2e (9) Deserializaion Error: no output from command 02:13:13 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure tity containers but only 0 are assigned" "attempt"=21 "retryIn"="1m0s" "elapsed"="20m0s" "ts"="2026-04-09 03:19:42.382195" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=22 "retryIn"="1m0s" "elapsed"="21m0s" "ts"="2026-04-09 03:20:42.384493" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=23 "retryIn"="1m0s" "elapsed"="22m0s" "ts"="2026-04-09 03:21:42.387621" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=24 "retryIn"="1m0s" "elapsed"="23m0s" "ts"="2026-04-09 03:22:42.391394" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=25 "retryIn"="1m0s" "elapsed"="24m0s" "ts"="2026-04-09 03:23:42.393047" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=26 "retryIn"="1m0s" "elapsed"="25m0s" "ts"="2026-04-09 03:24:42.397053" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=27 "retryIn"="1m0s" "elapsed"="26m0s" "ts"="2026-04-09 03:25:42.399495" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=28 "retryIn"="1m0s" "elapsed"="27m0s" "ts"="2026-04-09 03:26:42.402131" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=29 "retryIn"="1m0s" "elapsed"="28m0s" "ts"="2026-04-09 03:27:42.404280" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=30 "retryIn"="1m0s" "elapsed"="29m0s" "ts"="2026-04-09 03:28:42.406960" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=31 "retryIn"="1m0s" "elapsed"="30m0s" "ts"="2026-04-09 03:29:42.408890" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=32 "retryIn"="1m0s" "elapsed"="31m0s" "ts"="2026-04-09 03:30:42.411369" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=33 "retryIn"="1m0s" "elapsed"="32m0s" "ts"="2026-04-09 03:31:42.412850" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=34 "retryIn"="1m0s" "elapsed"="33m0s" "ts"="2026-04-09 03:32:42.421462" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=35 "elapsed"="34m0s" STEP: checking if the region supports availability zones @ 04/09/26 03:32:42.421 STEP: creating a resource group @ 04/09/26 03:32:44.196 "ts"="2026-04-09 03:32:44.196154" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-6svcw6" STEP: creating cluster parameters @ 04/09/26 03:32:47.439 STEP: creating customer resources @ 04/09/26 03:32:47.439 STEP: creating the HCP cluster @ 04/09/26 03:33:26.766 "ts"="2026-04-09 03:33:26.766493" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-6svcw6" STEP: verifying the cluster has default autoscaling parameters @ 04/09/26 03:43:59.519 STEP: getting admin credentials for the cluster @ 04/09/26 03:44:05.57 STEP: creating the AZ nodepool with 500 max replicas @ 04/09/26 03:46:41.044 STEP: verifying the AZ nodepool has the correct autoscaling configuration @ 04/09/26 04:06:10.866 STEP: verifying node count after AZ nodepool creation @ 04/09/26 04:06:11.096 STEP: updating the AZ nodepool max replicas from 500 to 4 before creating the next nodepool @ 04/09/26 04:06:11.701 STEP: verifying the AZ nodepool max replicas was updated to 4 @ 04/09/26 04:10:03.502 STEP: creating the no-AZ nodepool with 200 max replicas @ 04/09/26 04:10:03.786 Command Error: signal: killed Deserializaion Error: no output from command tity containers but only 0 are assigned" "attempt"=21 "retryIn"="1m0s" "elapsed"="20m0s" "ts"="2026-04-09 03:19:42.382195" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=22 "retryIn"="1m0s" "elapsed"="21m0s" "ts"="2026-04-09 03:20:42.384493" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=23 "retryIn"="1m0s" "elapsed"="22m0s" "ts"="2026-04-09 03:21:42.387621" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=24 "retryIn"="1m0s" "elapsed"="23m0s" "ts"="2026-04-09 03:22:42.391394" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=25 "retryIn"="1m0s" "elapsed"="24m0s" "ts"="2026-04-09 03:23:42.393047" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=26 "retryIn"="1m0s" "elapsed"="25m0s" "ts"="2026-04-09 03:24:42.397053" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=27 "retryIn"="1m0s" "elapsed"="26m0s" "ts"="2026-04-09 03:25:42.399495" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=28 "retryIn"="1m0s" "elapsed"="27m0s" "ts"="2026-04-09 03:26:42.402131" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=29 "retryIn"="1m0s" "elapsed"="28m0s" "ts"="2026-04-09 03:27:42.404280" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=30 "retryIn"="1m0s" "elapsed"="29m0s" "ts"="2026-04-09 03:28:42.406960" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=31 "retryIn"="1m0s" "elapsed"="30m0s" "ts"="2026-04-09 03:29:42.408890" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=32 "retryIn"="1m0s" "elapsed"="31m0s" "ts"="2026-04-09 03:30:42.411369" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=33 "retryIn"="1m0s" "elapsed"="32m0s" "ts"="2026-04-09 03:31:42.412850" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=34 "retryIn"="1m0s" "elapsed"="33m0s" "ts"="2026-04-09 03:32:42.421462" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=35 "elapsed"="34m0s" STEP: checking if the region supports availability zones @ 04/09/26 03:32:42.421 STEP: creating a resource group @ 04/09/26 03:32:44.196 "ts"="2026-04-09 03:32:44.196154" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-6svcw6" STEP: creating cluster parameters @ 04/09/26 03:32:47.439 STEP: creating customer resources @ 04/09/26 03:32:47.439 STEP: creating the HCP cluster @ 04/09/26 03:33:26.766 "ts"="2026-04-09 03:33:26.766493" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-6svcw6" STEP: verifying the cluster has default autoscaling parameters @ 04/09/26 03:43:59.519 STEP: getting admin credentials for the cluster @ 04/09/26 03:44:05.57 STEP: creating the AZ nodepool with 500 max replicas @ 04/09/26 03:46:41.044 STEP: verifying the AZ nodepool has the correct autoscaling configuration @ 04/09/26 04:06:10.866 STEP: verifying node count after AZ nodepool creation @ 04/09/26 04:06:11.096 STEP: updating the AZ nodepool max replicas from 500 to 4 before creating the next nodepool @ 04/09/26 04:06:11.701 STEP: verifying the AZ nodepool max replicas was updated to 4 @ 04/09/26 04:10:03.502 STEP: creating the no-AZ nodepool with 200 max replicas @ 04/09/26 04:10:03.786 Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 02:13:13 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc000620120>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2r9cp8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2r9cp8, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc000620120>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2r9cp8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2r9cp8, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 02:13:13 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure etup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169 "ts"="2026-04-09 03:19:22.190931" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-use-workload-identity-via-the-cluster-OIDC-issuer-URL\|pid:930" "ts"="2026-04-09 03:19:22.197971" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/09/26 03:19:22.198 "ts"="2026-04-09 03:19:22.198017" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-pvl2d2" STEP: creating cluster parameters @ 04/09/26 03:19:26.762 STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/09/26 03:19:26.762 STEP: creating the HCP cluster @ 04/09/26 03:20:04.902 "ts"="2026-04-09 03:20:04.903044" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-pvl2d2" STEP: getting the cluster's OIDC issuer URL @ 04/09/26 03:40:23.92 Cluster OIDC issuer URL: https://uksouth.oic.aro-hcp.azure.com/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2picpq66ab9mj7d4bgfs5gp11o5je650 STEP: getting admin credentials @ 04/09/26 03:40:26.284 STEP: ensuring the cluster is viable @ 04/09/26 03:43:37.288 STEP: creating the node pool @ 04/09/26 03:43:37.893 STEP: verifying nodes are ready @ 04/09/26 03:56:28.271 STEP: creating a user-assigned managed identity for the test @ 04/09/26 03:56:28.871 Created managed identity with client ID: cf369737-e848-46a4-ab5c-6d8bd20486a3 STEP: creating a federated identity credential for the service account @ 04/09/26 03:56:32.436 Created federated identity credential with issuer: https://uksouth.oic.aro-hcp.azure.com/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2picpq66ab9mj7d4bgfs5gp11o5je650, subject: system:serviceaccount:e2e-oidc-wi:wi-test-sa STEP: creating the test namespace and service account in the cluster @ 04/09/26 03:56:35.234 STEP: waiting for namespace UID range annotation to be set by the cluster-policy-controller @ 04/09/26 03:56:35.428 STEP: creating a pod that authenticates to Azure using federated workload identity credentials @ 04/09/26 03:56:40.62 I0409 03:56:40.836741 930 warnings.go:110] "Warning: would violate PodSecurity \"restricted:latest\": runAsNonRoot != true (pod or container \"azure-cli\" must set securityContext.runAsNonRoot=true), runAsUser=0 (container \"azure-cli\" must not set runAsUser=0)" STEP: waiting for the pod to complete successfully @ 04/09/26 03:56:40.836 Pod state transitioned from 'Pending' to 'Running' Pod state transitioned from 'Running' to 'Succeeded' Workload identity authentication succeeded. Pod completed successfully. "ts"="2026-04-09 03:57:31.424893" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 03:57:31.425059" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-oidc-wi-cluster-wfpm58" "resourceGroup"="oidc-wi-pvl2d2" "ts"="2026-04-09 03:57:31.783064" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-oidc-wi-cluster-wfpm58" "resourceGroup"="" "ts"="2026-04-09 03:57:33.276552" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz" "ts"="2026-04-09 03:57:33.276672" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz" "ts"="2026-04-09 03:57:33.276738" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 03:57:33.278253" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-71" "ts"="2026-04-09 03:57:33.278266" "level"=0 "msg"="collecting deployments" "resourceGroup"="oidc-wi-pvl2d2" "ts"="2026-04-09 03:57:33.416422" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-71" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 03:57:33.540480" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-71" "deployment"="msi-deployment-71" "ts"="2026-04-09 03:57:33.829957" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-pvl2d2" "deployment"="pooledNonMsiScopedAssignments-s6kwp7xg43dbi" "ts"="2026-04-09 03:57:33.973325" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-pvl2d2" "deployment"="customer-infra-oidc-wi-cluster-wfpm58" "ts"="2026-04-09 03:57:34.121191" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 03:57:34.121312" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 03:57:34.121454" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="oidc-wi-pvl2d2" Command Error: signal: killed Deserializaion Error: no output from command etup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169 "ts"="2026-04-09 03:19:22.190931" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-use-workload-identity-via-the-cluster-OIDC-issuer-URL\|pid:930" "ts"="2026-04-09 03:19:22.197971" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/09/26 03:19:22.198 "ts"="2026-04-09 03:19:22.198017" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-pvl2d2" STEP: creating cluster parameters @ 04/09/26 03:19:26.762 STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/09/26 03:19:26.762 STEP: creating the HCP cluster @ 04/09/26 03:20:04.902 "ts"="2026-04-09 03:20:04.903044" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-pvl2d2" STEP: getting the cluster's OIDC issuer URL @ 04/09/26 03:40:23.92 Cluster OIDC issuer URL: https://uksouth.oic.aro-hcp.azure.com/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2picpq66ab9mj7d4bgfs5gp11o5je650 STEP: getting admin credentials @ 04/09/26 03:40:26.284 STEP: ensuring the cluster is viable @ 04/09/26 03:43:37.288 STEP: creating the node pool @ 04/09/26 03:43:37.893 STEP: verifying nodes are ready @ 04/09/26 03:56:28.271 STEP: creating a user-assigned managed identity for the test @ 04/09/26 03:56:28.871 Created managed identity with client ID: cf369737-e848-46a4-ab5c-6d8bd20486a3 STEP: creating a federated identity credential for the service account @ 04/09/26 03:56:32.436 Created federated identity credential with issuer: https://uksouth.oic.aro-hcp.azure.com/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/2picpq66ab9mj7d4bgfs5gp11o5je650, subject: system:serviceaccount:e2e-oidc-wi:wi-test-sa STEP: creating the test namespace and service account in the cluster @ 04/09/26 03:56:35.234 STEP: waiting for namespace UID range annotation to be set by the cluster-policy-controller @ 04/09/26 03:56:35.428 STEP: creating a pod that authenticates to Azure using federated workload identity credentials @ 04/09/26 03:56:40.62 I0409 03:56:40.836741 930 warnings.go:110] "Warning: would violate PodSecurity \"restricted:latest\": runAsNonRoot != true (pod or container \"azure-cli\" must set securityContext.runAsNonRoot=true), runAsUser=0 (container \"azure-cli\" must not set runAsUser=0)" STEP: waiting for the pod to complete successfully @ 04/09/26 03:56:40.836 Pod state transitioned from 'Pending' to 'Running' Pod state transitioned from 'Running' to 'Succeeded' Workload identity authentication succeeded. Pod completed successfully. "ts"="2026-04-09 03:57:31.424893" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 03:57:31.425059" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-oidc-wi-cluster-wfpm58" "resourceGroup"="oidc-wi-pvl2d2" "ts"="2026-04-09 03:57:31.783064" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-oidc-wi-cluster-wfpm58" "resourceGroup"="" "ts"="2026-04-09 03:57:33.276552" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz" "ts"="2026-04-09 03:57:33.276672" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-f2883517cea1430b4915f04acc052b067a0c30de1b37a57f443ec7d05cd4dc52.yaml.gz" "ts"="2026-04-09 03:57:33.276738" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 03:57:33.278253" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-71" "ts"="2026-04-09 03:57:33.278266" "level"=0 "msg"="collecting deployments" "resourceGroup"="oidc-wi-pvl2d2" "ts"="2026-04-09 03:57:33.416422" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-71" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 03:57:33.540480" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-71" "deployment"="msi-deployment-71" "ts"="2026-04-09 03:57:33.829957" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-pvl2d2" "deployment"="pooledNonMsiScopedAssignments-s6kwp7xg43dbi" "ts"="2026-04-09 03:57:33.973325" "level"=0 "msg"="collecting operations" "resourceGroup"="oidc-wi-pvl2d2" "deployment"="customer-infra-oidc-wi-cluster-wfpm58" "ts"="2026-04-09 03:57:34.121191" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 03:57:34.121312" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 03:57:34.121454" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="oidc-wi-pvl2d2" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:13:13 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775704466 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 W0409 03:14:26.771406 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0409 03:14:26.888642 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-09 03:14:26.889032" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:893" "ts"="2026-04-09 03:14:26.895878" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/09/26 03:14:26.895 "ts"="2026-04-09 03:14:26.895962" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-rtdzxc-rgpqsm" STEP: creating cluster parameters at control plane version @ 04/09/26 03:14:30.733 W0409 03:14:30.812060 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/09/26 03:14:30.812 STEP: creating the HCP cluster with version 4.21.7 @ 04/09/26 03:15:20.256 "ts"="2026-04-09 03:15:20.256746" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-rtdzxc" "resourceGroup"="rg-np-version-upgrade-rtdzxc-rgpqsm" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/09/26 03:39:18.605 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/09/26 03:53:11.727 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/09/26 04:01:25.78 W0409 04:01:25.907854 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/09/26 04:01:25.907 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/09/26 04:01:26.096 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/09/26 04:02:42.093 Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775704466 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 W0409 03:14:26.771406 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0409 03:14:26.888642 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-09 03:14:26.889032" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:893" "ts"="2026-04-09 03:14:26.895878" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/09/26 03:14:26.895 "ts"="2026-04-09 03:14:26.895962" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-rtdzxc-rgpqsm" STEP: creating cluster parameters at control plane version @ 04/09/26 03:14:30.733 W0409 03:14:30.812060 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/09/26 03:14:30.812 STEP: creating the HCP cluster with version 4.21.7 @ 04/09/26 03:15:20.256 "ts"="2026-04-09 03:15:20.256746" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-rtdzxc" "resourceGroup"="rg-np-version-upgrade-rtdzxc-rgpqsm" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/09/26 03:39:18.605 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/09/26 03:53:11.727 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/09/26 04:01:25.78 W0409 04:01:25.907854 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/09/26 04:01:25.907 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/09/26 04:01:26.096 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/09/26 04:02:42.093 Command Error: signal: killed Deserializaion Error: no output from command Interrupted by User 02:13:13 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User Deserializaion Error: no output from command 02:13:13 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure -fsmm4" "labels"={"hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "failure-domain.beta.kubernetes.io/region"="uksouth" "hypershift.openshift.io/nodePool"="k5s4w5q8w1f4e5m-np-1" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="k5s4w5q8w1f4e5m-np-1-vs69q-fsmm4" "node-role.kubernetes.io/worker"="" "kubernetes.io/os"="linux" "topology.kubernetes.io/region"="uksouth" "beta.kubernetes.io/arch"="amd64" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="0"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-09 04:13:15.242897" "level"=0 "msg"="Node" "name"="k5s4w5q8w1f4e5m-np-1-vs69q-k6cjg" "labels"={"beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="uksouth" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/arch"="amd64" "kubernetes.io/os"="linux" "kubernetes.io/hostname"="k5s4w5q8w1f4e5m-np-1-vs69q-k6cjg" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/nodePool"="k5s4w5q8w1f4e5m-np-1" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "key1"="value1" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="uksouth"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-09 04:13:15.242918" "level"=0 "msg"="Node" "name"="k5s4w5q8w1f4e5m-np-1-vs69q-lwmnr" "labels"={"beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="1" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="uksouth" "hypershift.openshift.io/nodePool"="k5s4w5q8w1f4e5m-np-1" "key2"="value2" "node-role.kubernetes.io/worker"="" "topology.kubernetes.io/region"="uksouth" "topology.kubernetes.io/zone"="1" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/arch"="amd64" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="k5s4w5q8w1f4e5m-np-1-vs69q-lwmnr" "topology.disk.csi.azure.com/zone"=""} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/09/26 04:13:15.242 STEP: verifying original taints persist on 2 nodes @ 04/09/26 04:13:15.243 "ts"="2026-04-09 04:13:15.243171" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:13:15.243246" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-2hzkp9" "resourceGroup"="rg-np-labels-taints-fs4hs8" "ts"="2026-04-09 04:13:15.414442" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-2hzkp9" "resourceGroup"="" "ts"="2026-04-09 04:13:16.509458" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-09 04:13:16.509599" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-09 04:13:16.509657" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:13:16.511230" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "ts"="2026-04-09 04:13:16.511263" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-fs4hs8" "ts"="2026-04-09 04:13:16.668175" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fs4hs8" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-09 04:13:16.673052" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:13:16.799826" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fs4hs8" "deployment"="customer-infra-cluster-np-labels-taints-2hzkp9" "ts"="2026-04-09 04:13:16.834046" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="msi-deployment-4" "ts"="2026-04-09 04:13:16.972023" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:13:16.972168" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:13:16.972298" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-fs4hs8" Command Error: signal: killed Deserializaion Error: no output from command -fsmm4" "labels"={"hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "failure-domain.beta.kubernetes.io/region"="uksouth" "hypershift.openshift.io/nodePool"="k5s4w5q8w1f4e5m-np-1" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="k5s4w5q8w1f4e5m-np-1-vs69q-fsmm4" "node-role.kubernetes.io/worker"="" "kubernetes.io/os"="linux" "topology.kubernetes.io/region"="uksouth" "beta.kubernetes.io/arch"="amd64" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="0"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-09 04:13:15.242897" "level"=0 "msg"="Node" "name"="k5s4w5q8w1f4e5m-np-1-vs69q-k6cjg" "labels"={"beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="uksouth" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/arch"="amd64" "kubernetes.io/os"="linux" "kubernetes.io/hostname"="k5s4w5q8w1f4e5m-np-1-vs69q-k6cjg" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/nodePool"="k5s4w5q8w1f4e5m-np-1" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "key1"="value1" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="uksouth"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-09 04:13:15.242918" "level"=0 "msg"="Node" "name"="k5s4w5q8w1f4e5m-np-1-vs69q-lwmnr" "labels"={"beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="1" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="uksouth" "hypershift.openshift.io/nodePool"="k5s4w5q8w1f4e5m-np-1" "key2"="value2" "node-role.kubernetes.io/worker"="" "topology.kubernetes.io/region"="uksouth" "topology.kubernetes.io/zone"="1" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/arch"="amd64" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="k5s4w5q8w1f4e5m-np-1-vs69q-lwmnr" "topology.disk.csi.azure.com/zone"=""} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/09/26 04:13:15.242 STEP: verifying original taints persist on 2 nodes @ 04/09/26 04:13:15.243 "ts"="2026-04-09 04:13:15.243171" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:13:15.243246" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-2hzkp9" "resourceGroup"="rg-np-labels-taints-fs4hs8" "ts"="2026-04-09 04:13:15.414442" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-2hzkp9" "resourceGroup"="" "ts"="2026-04-09 04:13:16.509458" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-09 04:13:16.509599" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-09 04:13:16.509657" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:13:16.511230" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "ts"="2026-04-09 04:13:16.511263" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-fs4hs8" "ts"="2026-04-09 04:13:16.668175" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fs4hs8" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-09 04:13:16.673052" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:13:16.799826" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fs4hs8" "deployment"="customer-infra-cluster-np-labels-taints-2hzkp9" "ts"="2026-04-09 04:13:16.834046" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="msi-deployment-4" "ts"="2026-04-09 04:13:16.972023" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:13:16.972168" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:13:16.972298" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-fs4hs8" Command Error: signal: killed Deserializaion Error: no output from command tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> 02:13:13 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:168]: Unexpected error: <url.Error \| 0xc001248ab0>: Get "https://api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.q6i1z1a9v3j3w8r.qgw8.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:168]: Unexpected error: <url.Error \| 0xc001248ab0>: Get "https://api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.q6i1z1a9v3j3w8r.qgw8.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io ... occurred tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> 02:13:13 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error: <errors.joinError \| 0xc0013be210>: VerifyImageRegistryDisabled failed: wrong type of error: url.Error, Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io VerifyBasicAccess failed: failed to list services: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error: <errors.joinError \| 0xc0013be210>: VerifyImageRegistryDisabled failed: wrong type of error: url.Error, Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io VerifyBasicAccess failed: failed to list services: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io ... occurred Interrupted by User 02:13:13 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User unknown (1) Process did not finish before 2h0m0s timeout 02:13:13 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-09T04:15:36Z"}
02:00:33	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	11	e2e (10) Deserializaion Error: no output from command 02:00:33 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure 1 identity containers but only 0 are assigned" "attempt"=9 "retryIn"="1m0s" "elapsed"="8m0s" "ts"="2026-04-09 03:06:10.991148" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=10 "retryIn"="1m0s" "elapsed"="9m0s" "ts"="2026-04-09 03:07:10.993390" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=11 "retryIn"="1m0s" "elapsed"="10m0s" "ts"="2026-04-09 03:08:10.995597" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=12 "retryIn"="1m0s" "elapsed"="11m0s" "ts"="2026-04-09 03:09:10.998072" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=13 "retryIn"="1m0s" "elapsed"="12m0s" "ts"="2026-04-09 03:10:10.999567" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=14 "retryIn"="1m0s" "elapsed"="13m0s" "ts"="2026-04-09 03:11:11.001831" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=15 "retryIn"="1m0s" "elapsed"="14m0s" "ts"="2026-04-09 03:12:11.003605" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=16 "retryIn"="1m0s" "elapsed"="15m0s" "ts"="2026-04-09 03:13:11.005877" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=17 "retryIn"="1m0s" "elapsed"="16m0s" "ts"="2026-04-09 03:14:11.007936" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=18 "retryIn"="1m0s" "elapsed"="17m0s" "ts"="2026-04-09 03:15:11.697320" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=19 "retryIn"="1m0s" "elapsed"="18m1s" "ts"="2026-04-09 03:16:11.699773" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=20 "retryIn"="1m0s" "elapsed"="19m1s" "ts"="2026-04-09 03:17:11.702809" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=21 "retryIn"="1m0s" "elapsed"="20m1s" "ts"="2026-04-09 03:18:11.705098" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=22 "retryIn"="1m0s" "elapsed"="21m1s" "ts"="2026-04-09 03:19:11.715330" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=23 "elapsed"="22m1s" STEP: checking if the region supports availability zones @ 04/09/26 03:19:11.715 STEP: creating a resource group @ 04/09/26 03:19:13.671 "ts"="2026-04-09 03:19:13.671663" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-dkwlpg" STEP: creating cluster parameters @ 04/09/26 03:19:17.02 STEP: creating customer resources @ 04/09/26 03:19:17.02 STEP: creating the HCP cluster @ 04/09/26 03:20:05.95 "ts"="2026-04-09 03:20:05.950751" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-dkwlpg" STEP: verifying the cluster has default autoscaling parameters @ 04/09/26 03:37:52.738 STEP: getting admin credentials for the cluster @ 04/09/26 03:37:53.37 STEP: creating the AZ nodepool with 500 max replicas @ 04/09/26 03:44:04.67 STEP: verifying the AZ nodepool has the correct autoscaling configuration @ 04/09/26 03:53:13.983 STEP: verifying node count after AZ nodepool creation @ 04/09/26 03:53:14.197 STEP: updating the AZ nodepool max replicas from 500 to 4 before creating the next nodepool @ 04/09/26 03:53:14.727 STEP: verifying the AZ nodepool max replicas was updated to 4 @ 04/09/26 03:59:30.841 STEP: creating the no-AZ nodepool with 200 max replicas @ 04/09/26 03:59:31.163 Command Error: signal: killed Deserializaion Error: no output from command 1 identity containers but only 0 are assigned" "attempt"=9 "retryIn"="1m0s" "elapsed"="8m0s" "ts"="2026-04-09 03:06:10.991148" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=10 "retryIn"="1m0s" "elapsed"="9m0s" "ts"="2026-04-09 03:07:10.993390" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=11 "retryIn"="1m0s" "elapsed"="10m0s" "ts"="2026-04-09 03:08:10.995597" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=12 "retryIn"="1m0s" "elapsed"="11m0s" "ts"="2026-04-09 03:09:10.998072" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=13 "retryIn"="1m0s" "elapsed"="12m0s" "ts"="2026-04-09 03:10:10.999567" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=14 "retryIn"="1m0s" "elapsed"="13m0s" "ts"="2026-04-09 03:11:11.001831" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=15 "retryIn"="1m0s" "elapsed"="14m0s" "ts"="2026-04-09 03:12:11.003605" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=16 "retryIn"="1m0s" "elapsed"="15m0s" "ts"="2026-04-09 03:13:11.005877" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=17 "retryIn"="1m0s" "elapsed"="16m0s" "ts"="2026-04-09 03:14:11.007936" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=18 "retryIn"="1m0s" "elapsed"="17m0s" "ts"="2026-04-09 03:15:11.697320" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=19 "retryIn"="1m0s" "elapsed"="18m1s" "ts"="2026-04-09 03:16:11.699773" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=20 "retryIn"="1m0s" "elapsed"="19m1s" "ts"="2026-04-09 03:17:11.702809" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=21 "retryIn"="1m0s" "elapsed"="20m1s" "ts"="2026-04-09 03:18:11.705098" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=22 "retryIn"="1m0s" "elapsed"="21m1s" "ts"="2026-04-09 03:19:11.715330" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=23 "elapsed"="22m1s" STEP: checking if the region supports availability zones @ 04/09/26 03:19:11.715 STEP: creating a resource group @ 04/09/26 03:19:13.671 "ts"="2026-04-09 03:19:13.671663" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-dkwlpg" STEP: creating cluster parameters @ 04/09/26 03:19:17.02 STEP: creating customer resources @ 04/09/26 03:19:17.02 STEP: creating the HCP cluster @ 04/09/26 03:20:05.95 "ts"="2026-04-09 03:20:05.950751" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-dkwlpg" STEP: verifying the cluster has default autoscaling parameters @ 04/09/26 03:37:52.738 STEP: getting admin credentials for the cluster @ 04/09/26 03:37:53.37 STEP: creating the AZ nodepool with 500 max replicas @ 04/09/26 03:44:04.67 STEP: verifying the AZ nodepool has the correct autoscaling configuration @ 04/09/26 03:53:13.983 STEP: verifying node count after AZ nodepool creation @ 04/09/26 03:53:14.197 STEP: updating the AZ nodepool max replicas from 500 to 4 before creating the next nodepool @ 04/09/26 03:53:14.727 STEP: verifying the AZ nodepool max replicas was updated to 4 @ 04/09/26 03:59:30.841 STEP: creating the no-AZ nodepool with 200 max replicas @ 04/09/26 03:59:31.163 Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:00:33 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure ================= Random Seed: 1775703951 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237 W0409 03:05:51.455222 904 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0409 03:05:51.559512 904 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-09 03:05:51.559836" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest\|pid:904" "ts"="2026-04-09 03:05:51.567639" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/09/26 03:05:51.567 "ts"="2026-04-09 03:05:51.567727" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" STEP: creating cluster parameters at control plane version @ 04/09/26 03:05:55.822 STEP: creating customer resources @ 04/09/26 03:05:55.926 STEP: creating the HCP cluster with version 4.20.17 @ 04/09/26 03:06:44.311 "ts"="2026-04-09 03:06:44.311574" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-ql4mqn" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/09/26 03:17:47.84 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/09/26 03:29:40.742 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/09/26 03:39:15.385 STEP: capturing node release images before upgrade @ 04/09/26 03:39:15.564 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/09/26 03:39:15.754 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/09/26 03:40:29.087 STEP: verifying node pool GET still reflects the new version @ 04/09/26 04:02:35.07 "ts"="2026-04-09 04:02:38.256269" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:02:38.256359" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-ql4mqn-l6q8c7" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "ts"="2026-04-09 04:02:38.425554" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-ql4mqn-l6q8c7" "resourceGroup"="" "ts"="2026-04-09 04:02:40.358398" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-09 04:02:40.358532" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-09 04:02:40.358597" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:02:40.360177" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-46" "ts"="2026-04-09 04:02:40.360200" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "ts"="2026-04-09 04:02:40.508440" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-46" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:02:40.542981" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "deployment"="pooledNonMsiScopedAssignments-cv7d7g7burna6" "ts"="2026-04-09 04:02:40.631200" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-46" "deployment"="msi-deployment-46" "ts"="2026-04-09 04:02:40.690043" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "deployment"="customer-infra-np-version-upgrade-cluster-ql4mqn-l6q8c7" "ts"="2026-04-09 04:02:40.862864" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:02:40.863018" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:02:40.863144" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" Command Error: signal: killed Deserializaion Error: no output from command ================= Random Seed: 1775703951 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237 W0409 03:05:51.455222 904 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0409 03:05:51.559512 904 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-09 03:05:51.559836" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest\|pid:904" "ts"="2026-04-09 03:05:51.567639" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/09/26 03:05:51.567 "ts"="2026-04-09 03:05:51.567727" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" STEP: creating cluster parameters at control plane version @ 04/09/26 03:05:55.822 STEP: creating customer resources @ 04/09/26 03:05:55.926 STEP: creating the HCP cluster with version 4.20.17 @ 04/09/26 03:06:44.311 "ts"="2026-04-09 03:06:44.311574" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-ql4mqn" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/09/26 03:17:47.84 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/09/26 03:29:40.742 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/09/26 03:39:15.385 STEP: capturing node release images before upgrade @ 04/09/26 03:39:15.564 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/09/26 03:39:15.754 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/09/26 03:40:29.087 STEP: verifying node pool GET still reflects the new version @ 04/09/26 04:02:35.07 "ts"="2026-04-09 04:02:38.256269" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:02:38.256359" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-ql4mqn-l6q8c7" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "ts"="2026-04-09 04:02:38.425554" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-ql4mqn-l6q8c7" "resourceGroup"="" "ts"="2026-04-09 04:02:40.358398" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-09 04:02:40.358532" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-09 04:02:40.358597" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:02:40.360177" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-46" "ts"="2026-04-09 04:02:40.360200" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "ts"="2026-04-09 04:02:40.508440" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-46" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:02:40.542981" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "deployment"="pooledNonMsiScopedAssignments-cv7d7g7burna6" "ts"="2026-04-09 04:02:40.631200" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-46" "deployment"="msi-deployment-46" "ts"="2026-04-09 04:02:40.690043" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" "deployment"="customer-infra-np-version-upgrade-cluster-ql4mqn-l6q8c7" "ts"="2026-04-09 04:02:40.862864" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:02:40.863018" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:02:40.863144" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-ql4mqn-hbrtb8" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 02:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error: <fmt.wrapErrors \| 0xc001578ae0>: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7nb8nk" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7nb8nk, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error: <fmt.wrapErrors \| 0xc001578ae0>: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7nb8nk" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-7nb8nk, error: context deadline exceeded ... occurred Interrupted by User 02:00:33 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User Deserializaion Error: no output from command 02:00:33 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure ning results: skipping unrecognized cluster condition type "PromQL" W0409 03:00:39.275674 879 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-09 03:00:39.275951" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:879" "ts"="2026-04-09 03:00:39.281953" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/09/26 03:00:39.282 "ts"="2026-04-09 03:00:39.282084" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" STEP: creating cluster parameters at control plane version @ 04/09/26 03:00:43.984 W0409 03:00:44.067664 879 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/09/26 03:00:44.067 STEP: creating the HCP cluster with version 4.21.7 @ 04/09/26 03:01:43.145 "ts"="2026-04-09 03:01:43.145732" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gxv6r5" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/09/26 03:14:34.834 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/09/26 03:29:40.461 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/09/26 03:39:19.406 W0409 03:39:19.533244 879 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/09/26 03:39:19.533 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/09/26 03:39:19.728 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/09/26 03:40:22.889 STEP: verifying node pool GET still reflects the new version @ 04/09/26 04:04:29.49 "ts"="2026-04-09 04:04:30.537024" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:04:30.537109" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-gxv6r5-2shb8x" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "ts"="2026-04-09 04:04:30.880368" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-gxv6r5-2shb8x" "resourceGroup"="" "ts"="2026-04-09 04:04:32.708555" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-09 04:04:32.708690" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-09 04:04:32.708757" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:04:32.710584" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-21" "ts"="2026-04-09 04:04:32.710604" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "ts"="2026-04-09 04:04:32.895588" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "deployment"="pooledNonMsiScopedAssignments-bpnyq4yptvztk" "ts"="2026-04-09 04:04:33.040592" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-21" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:04:33.201672" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "deployment"="customer-infra-np-version-upgrade-cluster-gxv6r5-2shb8x" "ts"="2026-04-09 04:04:33.213643" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-21" "deployment"="msi-deployment-21" "ts"="2026-04-09 04:04:33.375025" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:04:33.375136" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:04:33.375250" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" Command Error: signal: killed Deserializaion Error: no output from command ning results: skipping unrecognized cluster condition type "PromQL" W0409 03:00:39.275674 879 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-09 03:00:39.275951" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:879" "ts"="2026-04-09 03:00:39.281953" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/09/26 03:00:39.282 "ts"="2026-04-09 03:00:39.282084" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" STEP: creating cluster parameters at control plane version @ 04/09/26 03:00:43.984 W0409 03:00:44.067664 879 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/09/26 03:00:44.067 STEP: creating the HCP cluster with version 4.21.7 @ 04/09/26 03:01:43.145 "ts"="2026-04-09 03:01:43.145732" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gxv6r5" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/09/26 03:14:34.834 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/09/26 03:29:40.461 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/09/26 03:39:19.406 W0409 03:39:19.533244 879 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/09/26 03:39:19.533 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/09/26 03:39:19.728 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/09/26 03:40:22.889 STEP: verifying node pool GET still reflects the new version @ 04/09/26 04:04:29.49 "ts"="2026-04-09 04:04:30.537024" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:04:30.537109" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-gxv6r5-2shb8x" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "ts"="2026-04-09 04:04:30.880368" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-gxv6r5-2shb8x" "resourceGroup"="" "ts"="2026-04-09 04:04:32.708555" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-09 04:04:32.708690" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-09 04:04:32.708757" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:04:32.710584" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-21" "ts"="2026-04-09 04:04:32.710604" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "ts"="2026-04-09 04:04:32.895588" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "deployment"="pooledNonMsiScopedAssignments-bpnyq4yptvztk" "ts"="2026-04-09 04:04:33.040592" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-21" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:04:33.201672" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" "deployment"="customer-infra-np-version-upgrade-cluster-gxv6r5-2shb8x" "ts"="2026-04-09 04:04:33.213643" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-21" "deployment"="msi-deployment-21" "ts"="2026-04-09 04:04:33.375025" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:04:33.375136" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:04:33.375250" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-gxv6r5-lslgt8" Command Error: signal: killed Deserializaion Error: no output from command get deployment router-default -n openshift-ingress: Get "<url> 02:00:33 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:180]: Timed out after 600.000s. router-default deployment logs should be fetchable Expected success, but got an error: <fmt.wrapError \| 0xc000d582a0>: get deployment router-default -n openshift-ingress: Get "https://api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io:443/apis/apps/v1/namespaces/openshift-ingress/deployments/router-default": tls: failed to verify certificate: x509: certificate is valid for api.o6n1s5y5u2a8q7c.wjok.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:180]: Timed out after 600.000s. router-default deployment logs should be fetchable Expected success, but got an error: <fmt.wrapError \| 0xc000d582a0>: get deployment router-default -n openshift-ingress: Get "https://api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io:443/apis/apps/v1/namespaces/openshift-ingress/deployments/router-default": tls: failed to verify certificate: x509: certificate is valid for api.o6n1s5y5u2a8q7c.wjok.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io ... Deserializaion Error: no output from command 02:00:33 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure dRequestContent\",\n \"message\": \"Invalid value: 201: must be less than or equal to 200\",\n \"target\": \"properties.replicas\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: creating a nodepool @ 04/09/26 03:44:13.396 STEP: attempting to update nodepool version to higher than cluster version @ 04/09/26 03:56:28.371 "ts"="2026-04-09 03:56:29.023343" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-wwtbdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n \"target\": \"properties.version.id\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: attempting to update immutable platform profile fields @ 04/09/26 03:56:29.023 "ts"="2026-04-09 03:56:29.647891" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-wwtbdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"4 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.vmSize\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.osDisk\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.availabilityZone\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-04-09 03:56:30.265128" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 03:56:30.265213" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-75dgnz" "resourceGroup"="rg-negative-tests-wwtbdt" "ts"="2026-04-09 03:56:30.421886" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-75dgnz" "resourceGroup"="" "ts"="2026-04-09 03:56:31.420106" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-09 03:56:31.420230" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-09 03:56:31.420289" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 03:56:31.421880" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "ts"="2026-04-09 03:56:31.421913" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-wwtbdt" "ts"="2026-04-09 03:56:31.572184" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 03:56:31.578091" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-wwtbdt" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-04-09 03:56:31.700962" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="msi-deployment-53" "ts"="2026-04-09 03:56:31.724961" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-wwtbdt" "deployment"="customer-infra-negative-tests-cluster-75dgnz" "ts"="2026-04-09 03:56:31.872303" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 03:56:31.872425" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 03:56:31.872555" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-wwtbdt" Command Error: signal: killed Deserializaion Error: no output from command dRequestContent\",\n \"message\": \"Invalid value: 201: must be less than or equal to 200\",\n \"target\": \"properties.replicas\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: creating a nodepool @ 04/09/26 03:44:13.396 STEP: attempting to update nodepool version to higher than cluster version @ 04/09/26 03:56:28.371 "ts"="2026-04-09 03:56:29.023343" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-wwtbdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n \"target\": \"properties.version.id\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: attempting to update immutable platform profile fields @ 04/09/26 03:56:29.023 "ts"="2026-04-09 03:56:29.647891" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-wwtbdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"4 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.vmSize\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.osDisk\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.availabilityZone\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-04-09 03:56:30.265128" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 03:56:30.265213" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-75dgnz" "resourceGroup"="rg-negative-tests-wwtbdt" "ts"="2026-04-09 03:56:30.421886" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-75dgnz" "resourceGroup"="" "ts"="2026-04-09 03:56:31.420106" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-09 03:56:31.420230" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-09 03:56:31.420289" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 03:56:31.421880" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "ts"="2026-04-09 03:56:31.421913" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-wwtbdt" "ts"="2026-04-09 03:56:31.572184" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 03:56:31.578091" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-wwtbdt" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-04-09 03:56:31.700962" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="msi-deployment-53" "ts"="2026-04-09 03:56:31.724961" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-wwtbdt" "deployment"="customer-infra-negative-tests-cluster-75dgnz" "ts"="2026-04-09 03:56:31.872303" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 03:56:31.872425" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 03:56:31.872555" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-wwtbdt" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 02:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc00157a420>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-7g7qrf" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-7g7qrf, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapErrors \| 0xc00157a420>: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-7g7qrf" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-7g7qrf, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 02:00:33 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure s but only 0 are assigned" "attempt"=1 "retryIn"="1m0s" "elapsed"="0s" "ts"="2026-04-09 03:16:11.746501" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=2 "retryIn"="1m0s" "elapsed"="1m0s" "ts"="2026-04-09 03:17:11.749675" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=3 "retryIn"="1m0s" "elapsed"="2m0s" "ts"="2026-04-09 03:18:11.752116" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=4 "retryIn"="1m0s" "elapsed"="3m0s" "ts"="2026-04-09 03:19:11.754307" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=5 "retryIn"="1m0s" "elapsed"="4m0s" "ts"="2026-04-09 03:20:11.756347" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=6 "retryIn"="1m0s" "elapsed"="5m0s" "ts"="2026-04-09 03:21:11.766667" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=7 "elapsed"="6m0s" STEP: creating a resource group @ 04/09/26 03:21:11.766 "ts"="2026-04-09 03:21:11.766746" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-6bnbg7" STEP: creating cluster parameters @ 04/09/26 03:21:16.342 STEP: creating customer resources @ 04/09/26 03:21:16.342 STEP: creating the HCP cluster @ 04/09/26 03:22:15.097 "ts"="2026-04-09 03:22:15.097474" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-6bnbg7" STEP: creating a storage account for shoebox logs @ 04/09/26 03:40:30.261 "ts"="2026-04-09 03:40:50.138070" "level"=0 "msg"="storage account created" "name"="shoeboxzk56nx" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-6bnbg7/providers/Microsoft.Storage/storageAccounts/shoeboxzk56nx" STEP: enabling diagnostic settings on the HCP cluster @ 04/09/26 03:40:50.138 "ts"="2026-04-09 03:41:12.296341" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/09/26 03:41:12.296 "ts"="2026-04-09 04:02:12.501062" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxzk56nx" "timestamp"="2026-04-09T04:02:12Z" "ts"="2026-04-09 04:02:12.501238" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:02:12.501321" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-5c55gc" "resourceGroup"="shoebox-cluster-6bnbg7" "ts"="2026-04-09 04:02:12.656846" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-5c55gc" "resourceGroup"="" "ts"="2026-04-09 04:02:13.973758" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-09 04:02:13.973873" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-09 04:02:13.973934" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:02:13.975374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-130" "ts"="2026-04-09 04:02:13.975384" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-6bnbg7" "ts"="2026-04-09 04:02:14.127141" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-130" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:02:14.132746" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6bnbg7" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-09 04:02:14.257408" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-130" "deployment"="msi-deployment-130" "ts"="2026-04-09 04:02:14.262704" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6bnbg7" "deployment"="customer-infra-shoebox-hcp-cluster-5c55gc" "ts"="2026-04-09 04:02:14.414204" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:02:14.414333" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:02:14.414469" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-6bnbg7" Command Error: signal: killed Deserializaion Error: no output from command s but only 0 are assigned" "attempt"=1 "retryIn"="1m0s" "elapsed"="0s" "ts"="2026-04-09 03:16:11.746501" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=2 "retryIn"="1m0s" "elapsed"="1m0s" "ts"="2026-04-09 03:17:11.749675" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=3 "retryIn"="1m0s" "elapsed"="2m0s" "ts"="2026-04-09 03:18:11.752116" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=4 "retryIn"="1m0s" "elapsed"="3m0s" "ts"="2026-04-09 03:19:11.754307" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=5 "retryIn"="1m0s" "elapsed"="4m0s" "ts"="2026-04-09 03:20:11.756347" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=6 "retryIn"="1m0s" "elapsed"="5m0s" "ts"="2026-04-09 03:21:11.766667" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=7 "elapsed"="6m0s" STEP: creating a resource group @ 04/09/26 03:21:11.766 "ts"="2026-04-09 03:21:11.766746" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-6bnbg7" STEP: creating cluster parameters @ 04/09/26 03:21:16.342 STEP: creating customer resources @ 04/09/26 03:21:16.342 STEP: creating the HCP cluster @ 04/09/26 03:22:15.097 "ts"="2026-04-09 03:22:15.097474" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-6bnbg7" STEP: creating a storage account for shoebox logs @ 04/09/26 03:40:30.261 "ts"="2026-04-09 03:40:50.138070" "level"=0 "msg"="storage account created" "name"="shoeboxzk56nx" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-6bnbg7/providers/Microsoft.Storage/storageAccounts/shoeboxzk56nx" STEP: enabling diagnostic settings on the HCP cluster @ 04/09/26 03:40:50.138 "ts"="2026-04-09 03:41:12.296341" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/09/26 03:41:12.296 "ts"="2026-04-09 04:02:12.501062" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxzk56nx" "timestamp"="2026-04-09T04:02:12Z" "ts"="2026-04-09 04:02:12.501238" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-09 04:02:12.501321" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-5c55gc" "resourceGroup"="shoebox-cluster-6bnbg7" "ts"="2026-04-09 04:02:12.656846" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-5c55gc" "resourceGroup"="" "ts"="2026-04-09 04:02:13.973758" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-09 04:02:13.973873" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-09 04:02:13.973934" "level"=0 "msg"="collecting debug info" "ts"="2026-04-09 04:02:13.975374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-130" "ts"="2026-04-09 04:02:13.975384" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-6bnbg7" "ts"="2026-04-09 04:02:14.127141" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-130" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-09 04:02:14.132746" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6bnbg7" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-09 04:02:14.257408" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-130" "deployment"="msi-deployment-130" "ts"="2026-04-09 04:02:14.262704" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6bnbg7" "deployment"="customer-infra-shoebox-hcp-cluster-5c55gc" "ts"="2026-04-09 04:02:14.414204" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-09 04:02:14.414333" "level"=0 "msg"="deleting created resources" "ts"="2026-04-09 04:02:14.414469" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-6bnbg7" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 02:00:33 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapErrors \| 0xc000b03170>: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2rlk9m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2rlk9m, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapErrors \| 0xc000b03170>: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2rlk9m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-2rlk9m, error: context deadline exceeded ... occurred unknown (1) Process did not finish before 2h0m0s timeout 02:00:33 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-09T04:05:03Z"}