CIHealth Run Log

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
20:29:03	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	7	e2e (7) timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:29:03 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc0007fd2f0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-mhtwts" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-mhtwts, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc0007fd2f0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-mhtwts" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-mhtwts, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:29:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc000dfc0f0>: failed waiting for hcpCluster="cluster-candidate-4-20-vl26hx" in resourcegroup="rg-candidate-4-20-lbfbj4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-candidate-4-20-vl26hx in resource group rg-candidate-4-20-lbfbj4, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc000dfc0f0>: failed waiting for hcpCluster="cluster-candidate-4-20-vl26hx" in resourcegroup="rg-candidate-4-20-lbfbj4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-candidate-4-20-vl26hx in resource group rg-candidate-4-20-lbfbj4, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 20:29:03 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error: <fmt.wrapErrors \| 0xc0015c05a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cw77vd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cw77vd, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error: <fmt.wrapErrors \| 0xc0015c05a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cw77vd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-cw77vd, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 20:29:03 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc00156eb10>: failed to create HCP cluster cluster-zstream-4-19-6mnfl6, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-6mnfl6 in resource group rg-zstream-upgrade-4-19-bjj5n4, error: failed waiting for cluster="cluster-zstream-4-19-6mnfl6" in resourcegroup="rg-zstream-upgrade-4-19-bjj5n4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-6mnfl6 in resource group rg-zstream-upgrade-4-19-bjj5n4, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc00156eb10>: failed to create HCP cluster cluster-zstream-4-19-6mnfl6, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-6mnfl6 in resource group rg-zstream-upgrade-4-19-bjj5n4, error: failed waiting for cluster="cluster-zstream-4-19-6mnfl6" in resourcegroup="rg-zstream-upgrade-4-19-bjj5n4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-19-6mnfl6 in resource group rg-zstream-upgrade-4-19-bjj5n4, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 20:29:03 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:239]: Unexpected error: <fmt.wrapError \| 0xc000c3a700>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-9hc929/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:239]: Unexpected error: <fmt.wrapError \| 0xc000c3a700>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-9hc929/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 20:29:03 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc000e73080>: failed waiting for hcpCluster="admin-cred-lifecycle-5wmv8g" in resourcegroup="admin-credential-lifecycle-test-46ctqx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wmv8g in resource group admin-credential-lifecycle-test-46ctqx, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc000e73080>: failed waiting for hcpCluster="admin-cred-lifecycle-5wmv8g" in resourcegroup="admin-credential-lifecycle-test-46ctqx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wmv8g in resource group admin-credential-lifecycle-test-46ctqx, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 20:29:03 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:116]: Unexpected error: <fmt.wrapErrors \| 0xc0012c4a20>: failed waiting for cluster="cilium-cluster" in resourcegroup="e2e-cilium-2xwn4s" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-2xwn4s, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:116]: Unexpected error: <fmt.wrapErrors \| 0xc0012c4a20>: failed waiting for cluster="cilium-cluster" in resourcegroup="e2e-cilium-2xwn4s" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group e2e-cilium-2xwn4s, error: context deadline exceeded ... occurred
19:22:53	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 19:22:53 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc00095d2c0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-ql4p2r" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-ql4p2r, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc00095d2c0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-ql4p2r" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-ql4p2r, error: context deadline exceeded ... occurred
16:55:28	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	13	e2e (13) timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc00092c4b0>: failed to create HCP cluster cluster-zstream-4-21-2rbk85, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2rbk85 in resource group rg-zstream-upgrade-4-21-zqf4kq, error: failed waiting for cluster="cluster-zstream-4-21-2rbk85" in resourcegroup="rg-zstream-upgrade-4-21-zqf4kq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2rbk85 in resource group rg-zstream-upgrade-4-21-zqf4kq, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc00092c4b0>: failed to create HCP cluster cluster-zstream-4-21-2rbk85, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2rbk85 in resource group rg-zstream-upgrade-4-21-zqf4kq, error: failed waiting for cluster="cluster-zstream-4-21-2rbk85" in resourcegroup="rg-zstream-upgrade-4-21-zqf4kq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-2rbk85 in resource group rg-zstream-upgrade-4-21-zqf4kq, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:28 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc0009d5ec0>: failed to create HCP cluster np-version-upgrade-cluster-bnsj87, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bnsj87 in resource group rg-np-version-upgrade-bnsj87-27s69m, error: failed waiting for cluster="np-version-upgrade-cluster-bnsj87" in resourcegroup="rg-np-version-upgrade-bnsj87-27s69m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bnsj87 in resource group rg-np-version-upgrade-bnsj87-27s69m, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc0009d5ec0>: failed to create HCP cluster np-version-upgrade-cluster-bnsj87, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bnsj87 in resource group rg-np-version-upgrade-bnsj87-27s69m, error: failed waiting for cluster="np-version-upgrade-cluster-bnsj87" in resourcegroup="rg-np-version-upgrade-bnsj87-27s69m" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-version-upgrade-cluster-bnsj87 in resource group rg-np-version-upgrade-bnsj87-27s69m, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 16:55:28 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc0015240c0>: failed waiting for hcpCluster="cluster-candidate-4-20-7bz6dn" in resourcegroup="rg-candidate-4-20-bwpkkz" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-candidate-4-20-7bz6dn in resource group rg-candidate-4-20-bwpkkz, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc0015240c0>: failed waiting for hcpCluster="cluster-candidate-4-20-7bz6dn" in resourcegroup="rg-candidate-4-20-bwpkkz" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-candidate-4-20-7bz6dn in resource group rg-candidate-4-20-bwpkkz, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 16:55:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0009241e0>: failed waiting for hcpCluster="cluster-zstream-4-19-pxx6tf" in resourcegroup="rg-zstream-upgrade-4-19-t99kbg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-pxx6tf in resource group rg-zstream-upgrade-4-19-t99kbg, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0009241e0>: failed waiting for hcpCluster="cluster-zstream-4-19-pxx6tf" in resourcegroup="rg-zstream-upgrade-4-19-t99kbg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-pxx6tf in resource group rg-zstream-upgrade-4-19-t99kbg, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:28 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: Unexpected error: <fmt.wrapErrors \| 0xc000985290>: failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-b54j8c, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-b54j8c" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-b54j8c, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: Unexpected error: <fmt.wrapErrors \| 0xc000985290>: failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-b54j8c, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-b54j8c" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-b54j8c, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 16:55:28 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc000b774d0>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-7sl9xl" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-7sl9xl, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc000b774d0>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-7sl9xl" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-7sl9xl, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:28 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error: <fmt.wrapErrors \| 0xc0009823c0>: failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-vtr9x4, error: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vtr9x4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-vtr9x4, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error: <fmt.wrapErrors \| 0xc0009823c0>: failed to create HCP cluster oidc-wi-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-vtr9x4, error: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vtr9x4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster oidc-wi-cluster in resource group oidc-wi-vtr9x4, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:28 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc0015640c0>: failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-xh7g84, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-xh7g84" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-xh7g84, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc0015640c0>: failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-xh7g84, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-xh7g84" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-xh7g84, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 16:55:28 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error: <fmt.wrapErrors \| 0xc000ed8990>: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-26vrlj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-26vrlj, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error: <fmt.wrapErrors \| 0xc000ed8990>: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-26vrlj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-26vrlj, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 16:55:28 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc0010ffb60>: failed waiting for hcpCluster="admin-cred-lifecycle-6z9xfq" in resourcegroup="admin-credential-lifecycle-test-nzvvl9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-6z9xfq in resource group admin-credential-lifecycle-test-nzvvl9, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc0010ffb60>: failed waiting for hcpCluster="admin-cred-lifecycle-6z9xfq" in resourcegroup="admin-credential-lifecycle-test-nzvvl9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-6z9xfq in resource group admin-credential-lifecycle-test-nzvvl9, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:28 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc0008df680>: failed to create HCP cluster shoebox-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-mrrj8n, error: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-mrrj8n" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-mrrj8n, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc0008df680>: failed to create HCP cluster shoebox-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-mrrj8n, error: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-mrrj8n" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-mrrj8n, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:28 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapErrors \| 0xc0013de780>: failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-whzfsl, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-whzfsl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-whzfsl, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapErrors \| 0xc0013de780>: failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-whzfsl, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-whzfsl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-whzfsl, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 16:55:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc000bda7b0>: failed waiting for hcpCluster="cluster-zstream-4-20-wq2lcq" in resourcegroup="rg-zstream-upgrade-4-20-nspgt8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-wq2lcq in resource group rg-zstream-upgrade-4-20-nspgt8, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc000bda7b0>: failed waiting for hcpCluster="cluster-zstream-4-20-wq2lcq" in resourcegroup="rg-zstream-upgrade-4-20-nspgt8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-wq2lcq in resource group rg-zstream-upgrade-4-20-nspgt8, error: context deadline exceeded ... occurred
16:55:01	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	10	e2e (10) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 16:55:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapError \| 0xc0009b6280>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-tflxwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapError \| 0xc0009b6280>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-tflxwt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 16:55:01 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapError \| 0xc000e90120>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-rn26mb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapError \| 0xc000e90120>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-rn26mb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc001442240>: failed to create HCP cluster cluster-zstream-4-21-w6h4l7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-w6h4l7 in resource group rg-zstream-upgrade-4-21-dcfn2j, error: failed waiting for cluster="cluster-zstream-4-21-w6h4l7" in resourcegroup="rg-zstream-upgrade-4-21-dcfn2j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-w6h4l7 in resource group rg-zstream-upgrade-4-21-dcfn2j, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc001442240>: failed to create HCP cluster cluster-zstream-4-21-w6h4l7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-w6h4l7 in resource group rg-zstream-upgrade-4-21-dcfn2j, error: failed waiting for cluster="cluster-zstream-4-21-w6h4l7" in resourcegroup="rg-zstream-upgrade-4-21-dcfn2j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-w6h4l7 in resource group rg-zstream-upgrade-4-21-dcfn2j, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:01 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc000ae0210>: failed to create HCP cluster shoebox-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-fc89tz, error: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-fc89tz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-fc89tz, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:103]: Unexpected error: <fmt.wrapErrors \| 0xc000ae0210>: failed to create HCP cluster shoebox-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-fc89tz, error: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-fc89tz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster shoebox-hcp-cluster in resource group shoebox-cluster-fc89tz, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:01 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117]: Unexpected error: <fmt.wrapErrors \| 0xc000dcec00>: failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-8w8qgx, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-8w8qgx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-8w8qgx, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117]: Unexpected error: <fmt.wrapErrors \| 0xc000dcec00>: failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-8w8qgx, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-8w8qgx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-8w8qgx, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc0015a00f0>: failed to create HCP cluster cluster-zstream-4-20-dgwgxd, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-dgwgxd in resource group rg-zstream-upgrade-4-20-5mx8l9, error: failed waiting for cluster="cluster-zstream-4-20-dgwgxd" in resourcegroup="rg-zstream-upgrade-4-20-5mx8l9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-dgwgxd in resource group rg-zstream-upgrade-4-20-5mx8l9, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error: <fmt.wrapErrors \| 0xc0015a00f0>: failed to create HCP cluster cluster-zstream-4-20-dgwgxd, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-dgwgxd in resource group rg-zstream-upgrade-4-20-5mx8l9, error: failed waiting for cluster="cluster-zstream-4-20-dgwgxd" in resourcegroup="rg-zstream-upgrade-4-20-5mx8l9" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-dgwgxd in resource group rg-zstream-upgrade-4-20-5mx8l9, error: context deadline exceeded ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000e6eb10>: failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-b5cpds, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-b5cpds" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-b5cpds, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000e6eb10>: failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-b5cpds, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-b5cpds" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-b5cpds, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 16:55:01 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapError \| 0xc00053ad20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-vf4zmt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-rd4xqc/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapError \| 0xc00053ad20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-vf4zmt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-rd4xqc/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 16:55:01 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc0010c1a40>: failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-cbtf79, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-cbtf79" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-cbtf79, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc0010c1a40>: failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-cbtf79, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-cbtf79" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-cbtf79, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 16:55:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapError \| 0xc001312300>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-26bkkw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-phh9x4/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapError \| 0xc001312300>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-26bkkw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-phh9x4/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred
10:59:58	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	5	e2e (4) Deserializaion Error: no output from command 10:59:58 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure t/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226 W0410 12:04:14.665210 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 12:04:14.748424 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 12:04:14.748740" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest\|pid:893" "ts"="2026-04-10 12:04:14.933427" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 12:04:14.933 "ts"="2026-04-10 12:04:14.933646" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" STEP: creating cluster parameters at control plane version @ 04/10/26 12:04:19.473 STEP: creating customer resources @ 04/10/26 12:04:19.579 STEP: creating the HCP cluster with version 4.20.17 @ 04/10/26 12:05:04.201 "ts"="2026-04-10 12:05:04.201160" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-jnm8s9" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:16:02.227 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:01.442 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:09.867 STEP: capturing node release images before upgrade @ 04/10/26 12:29:10.041 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/10/26 12:29:10.08 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:52.504 STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:54.086 "ts"="2026-04-10 12:51:54.286433" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:51:54.286512" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 12:51:54.379693" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"="" "ts"="2026-04-10 12:51:55.257214" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 12:51:55.257316" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 12:51:55.257391" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:51:55.259038" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "ts"="2026-04-10 12:51:55.259056" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 12:51:55.353380" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="pooledNonMsiScopedAssignments-jfl6npesxtst4" "ts"="2026-04-10 12:51:55.415329" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w" "ts"="2026-04-10 12:51:55.578837" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:51:55.709848" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="msi-deployment-129" "ts"="2026-04-10 12:51:55.860303" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:51:55.860430" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:51:55.860601" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 13:02:14.746040" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 13:02:14.746065" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" Command Error: signal: killed Deserializaion Error: no output from command t/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226 W0410 12:04:14.665210 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 12:04:14.748424 893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 12:04:14.748740" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest\|pid:893" "ts"="2026-04-10 12:04:14.933427" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 12:04:14.933 "ts"="2026-04-10 12:04:14.933646" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" STEP: creating cluster parameters at control plane version @ 04/10/26 12:04:19.473 STEP: creating customer resources @ 04/10/26 12:04:19.579 STEP: creating the HCP cluster with version 4.20.17 @ 04/10/26 12:05:04.201 "ts"="2026-04-10 12:05:04.201160" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-jnm8s9" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:16:02.227 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:01.442 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:09.867 STEP: capturing node release images before upgrade @ 04/10/26 12:29:10.041 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/10/26 12:29:10.08 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:52.504 STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:54.086 "ts"="2026-04-10 12:51:54.286433" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:51:54.286512" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 12:51:54.379693" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"="" "ts"="2026-04-10 12:51:55.257214" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 12:51:55.257316" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 12:51:55.257391" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:51:55.259038" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "ts"="2026-04-10 12:51:55.259056" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 12:51:55.353380" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="pooledNonMsiScopedAssignments-jfl6npesxtst4" "ts"="2026-04-10 12:51:55.415329" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w" "ts"="2026-04-10 12:51:55.578837" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:51:55.709848" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="msi-deployment-129" "ts"="2026-04-10 12:51:55.860303" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:51:55.860430" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:51:55.860601" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 13:02:14.746040" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "ts"="2026-04-10 13:02:14.746065" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 10:59:58 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775823245 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 12:14:05.070742" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:919" "ts"="2026-04-10 12:14:05.079100" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 12:14:05.079 "ts"="2026-04-10 12:14:05.079151" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-7pqpv2" STEP: creating cluster parameters @ 04/10/26 12:14:09.56 STEP: creating customer resources @ 04/10/26 12:14:09.56 STEP: creating the HCP cluster @ 04/10/26 12:14:54.459 "ts"="2026-04-10 12:14:54.459322" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-7pqpv2" STEP: creating a storage account for shoebox logs @ 04/10/26 12:29:05.918 "ts"="2026-04-10 12:29:25.513737" "level"=0 "msg"="storage account created" "name"="shoeboxfpqdwk" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7pqpv2/providers/Microsoft.Storage/storageAccounts/shoeboxfpqdwk" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 12:29:25.513 "ts"="2026-04-10 12:29:30.702291" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 12:29:30.702 "ts"="2026-04-10 12:55:30.837118" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxfpqdwk" "timestamp"="2026-04-10T12:55:30Z" "ts"="2026-04-10 12:55:30.837292" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:55:30.837387" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf" "resourceGroup"="shoebox-cluster-7pqpv2" "ts"="2026-04-10 12:55:30.988434" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-j9gxkf" "resourceGroup"="" "ts"="2026-04-10 12:55:32.339790" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:55:32.339933" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:55:32.340053" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:55:32.341960" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "ts"="2026-04-10 12:55:32.342015" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-7pqpv2" "ts"="2026-04-10 12:55:32.680479" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 12:55:32.749543" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf" "ts"="2026-04-10 12:55:32.952168" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:55:33.567562" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="msi-deployment-95" "ts"="2026-04-10 12:55:33.845873" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:55:33.846012" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:55:33.846126" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-7pqpv2" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775823245 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 12:14:05.070742" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:919" "ts"="2026-04-10 12:14:05.079100" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 12:14:05.079 "ts"="2026-04-10 12:14:05.079151" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-7pqpv2" STEP: creating cluster parameters @ 04/10/26 12:14:09.56 STEP: creating customer resources @ 04/10/26 12:14:09.56 STEP: creating the HCP cluster @ 04/10/26 12:14:54.459 "ts"="2026-04-10 12:14:54.459322" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-7pqpv2" STEP: creating a storage account for shoebox logs @ 04/10/26 12:29:05.918 "ts"="2026-04-10 12:29:25.513737" "level"=0 "msg"="storage account created" "name"="shoeboxfpqdwk" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7pqpv2/providers/Microsoft.Storage/storageAccounts/shoeboxfpqdwk" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 12:29:25.513 "ts"="2026-04-10 12:29:30.702291" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 12:29:30.702 "ts"="2026-04-10 12:55:30.837118" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxfpqdwk" "timestamp"="2026-04-10T12:55:30Z" "ts"="2026-04-10 12:55:30.837292" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:55:30.837387" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf" "resourceGroup"="shoebox-cluster-7pqpv2" "ts"="2026-04-10 12:55:30.988434" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-j9gxkf" "resourceGroup"="" "ts"="2026-04-10 12:55:32.339790" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:55:32.339933" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:55:32.340053" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:55:32.341960" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "ts"="2026-04-10 12:55:32.342015" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-7pqpv2" "ts"="2026-04-10 12:55:32.680479" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 12:55:32.749543" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf" "ts"="2026-04-10 12:55:32.952168" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:55:33.567562" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="msi-deployment-95" "ts"="2026-04-10 12:55:33.845873" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:55:33.846012" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:55:33.846126" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-7pqpv2" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 10:59:58 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 12:03:04.020611" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:880" "ts"="2026-04-10 12:03:04.028462" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 12:03:04.028 "ts"="2026-04-10 12:03:04.028545" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" STEP: creating cluster parameters at control plane version @ 04/10/26 12:03:08.089 W0410 12:03:08.168105 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 12:03:08.168 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 12:04:03.752 "ts"="2026-04-10 12:04:03.752222" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-wdq4gs" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:15:44.21 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:06.745 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:05.032 W0410 12:29:05.158022 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 12:29:05.158 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 12:29:05.194 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:48.069 STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:49.726 "ts"="2026-04-10 12:51:50.360556" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:51:50.360655" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 12:51:50.487211" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"="" "ts"="2026-04-10 12:51:51.762518" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 12:51:51.762617" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 12:51:51.762676" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:51:51.764204" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "ts"="2026-04-10 12:51:51.764225" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 12:51:51.859754" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="pooledNonMsiScopedAssignments-dpbn3ikyauobc" "ts"="2026-04-10 12:51:52.188516" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7" "ts"="2026-04-10 12:51:52.242487" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:51:52.779918" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="msi-deployment-131" "ts"="2026-04-10 12:51:52.917289" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:51:52.917406" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:51:52.917517" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 13:02:10.173344" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 13:02:10.173371" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" Command Error: signal: killed Deserializaion Error: no output from command API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 12:03:04.020611" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:880" "ts"="2026-04-10 12:03:04.028462" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 12:03:04.028 "ts"="2026-04-10 12:03:04.028545" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" STEP: creating cluster parameters at control plane version @ 04/10/26 12:03:08.089 W0410 12:03:08.168105 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 12:03:08.168 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 12:04:03.752 "ts"="2026-04-10 12:04:03.752222" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-wdq4gs" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:15:44.21 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:06.745 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:05.032 W0410 12:29:05.158022 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 12:29:05.158 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 12:29:05.194 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:48.069 STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:49.726 "ts"="2026-04-10 12:51:50.360556" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:51:50.360655" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 12:51:50.487211" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"="" "ts"="2026-04-10 12:51:51.762518" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 12:51:51.762617" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-10 12:51:51.762676" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:51:51.764204" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "ts"="2026-04-10 12:51:51.764225" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 12:51:51.859754" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="pooledNonMsiScopedAssignments-dpbn3ikyauobc" "ts"="2026-04-10 12:51:52.188516" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7" "ts"="2026-04-10 12:51:52.242487" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:51:52.779918" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="msi-deployment-131" "ts"="2026-04-10 12:51:52.917289" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:51:52.917406" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:51:52.917517" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 13:02:10.173344" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "ts"="2026-04-10 13:02:10.173371" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" Command Error: signal: killed Deserializaion Error: no output from command ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' has an external authentication pending deletion; provider Microsoft.RedHatOpenShift 10:59:58 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:294]: Unexpected error: <fmt.wrapError \| 0xc001508c80>: failed creating external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-n5j64n" for cluster="cidr-connectivity-test": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n5j64n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/externalAuths/ext-auth-cidr -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Cluster '2pj8n2ov52hfcbj1lfepdkk77i8o48uj' has an external authentication pending deletion" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:294]: Unexpected error: <fmt.wrapError \| 0xc001508c80>: failed creating external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-n5j64n" for cluster="cidr-connectivity-test": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n5j64n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/externalAuths/ext-auth-cidr -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Cluster '2pj8n2ov52hfcbj1lfepdkk77i8o48uj' has an external authentication pending deletion" } } -------------------------------------------------------------------------------- ... occurred unknown (1) Process did not finish before 2h0m0s timeout 10:59:58 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T13:03:33Z"}
10:57:56	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	9	e2e (8) Deserializaion Error: no output from command 10:57:56 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775823200 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 12:13:20.678047" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:920" "ts"="2026-04-10 12:13:20.684932" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 12:13:20.684 "ts"="2026-04-10 12:13:20.685003" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-cgrnxm" STEP: creating cluster parameters @ 04/10/26 12:13:25.393 STEP: creating customer resources @ 04/10/26 12:13:25.393 STEP: creating the HCP cluster @ 04/10/26 12:14:15.839 "ts"="2026-04-10 12:14:15.840100" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-cgrnxm" STEP: creating a storage account for shoebox logs @ 04/10/26 12:26:52.987 "ts"="2026-04-10 12:27:14.860715" "level"=0 "msg"="storage account created" "name"="shoeboxcv42l9" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cgrnxm/providers/Microsoft.Storage/storageAccounts/shoeboxcv42l9" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 12:27:14.86 "ts"="2026-04-10 12:27:16.973481" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 12:27:16.973 "ts"="2026-04-10 12:56:17.519891" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxcv42l9" "timestamp"="2026-04-10T12:56:17Z" "ts"="2026-04-10 12:56:17.520097" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:56:17.520196" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-kcfn5p" "resourceGroup"="shoebox-cluster-cgrnxm" "ts"="2026-04-10 12:56:18.474282" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-kcfn5p" "resourceGroup"="" "ts"="2026-04-10 12:56:19.927127" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:56:19.927234" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:56:19.927293" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:56:19.928736" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-138" "ts"="2026-04-10 12:56:19.928752" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-cgrnxm" "ts"="2026-04-10 12:56:20.089477" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-138" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:56:20.632817" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-138" "deployment"="msi-deployment-138" "ts"="2026-04-10 12:56:20.686097" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-cgrnxm" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 12:56:20.911100" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-cgrnxm" "deployment"="customer-infra-shoebox-hcp-cluster-kcfn5p" "ts"="2026-04-10 12:56:21.092180" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:56:21.092317" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:56:21.092433" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-cgrnxm" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775823200 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 12:13:20.678047" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:920" "ts"="2026-04-10 12:13:20.684932" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 12:13:20.684 "ts"="2026-04-10 12:13:20.685003" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-cgrnxm" STEP: creating cluster parameters @ 04/10/26 12:13:25.393 STEP: creating customer resources @ 04/10/26 12:13:25.393 STEP: creating the HCP cluster @ 04/10/26 12:14:15.839 "ts"="2026-04-10 12:14:15.840100" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-cgrnxm" STEP: creating a storage account for shoebox logs @ 04/10/26 12:26:52.987 "ts"="2026-04-10 12:27:14.860715" "level"=0 "msg"="storage account created" "name"="shoeboxcv42l9" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cgrnxm/providers/Microsoft.Storage/storageAccounts/shoeboxcv42l9" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 12:27:14.86 "ts"="2026-04-10 12:27:16.973481" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 12:27:16.973 "ts"="2026-04-10 12:56:17.519891" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxcv42l9" "timestamp"="2026-04-10T12:56:17Z" "ts"="2026-04-10 12:56:17.520097" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:56:17.520196" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-kcfn5p" "resourceGroup"="shoebox-cluster-cgrnxm" "ts"="2026-04-10 12:56:18.474282" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-kcfn5p" "resourceGroup"="" "ts"="2026-04-10 12:56:19.927127" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:56:19.927234" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 12:56:19.927293" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:56:19.928736" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-138" "ts"="2026-04-10 12:56:19.928752" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-cgrnxm" "ts"="2026-04-10 12:56:20.089477" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-138" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:56:20.632817" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-138" "deployment"="msi-deployment-138" "ts"="2026-04-10 12:56:20.686097" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-cgrnxm" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 12:56:20.911100" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-cgrnxm" "deployment"="customer-infra-shoebox-hcp-cluster-kcfn5p" "ts"="2026-04-10 12:56:21.092180" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:56:21.092317" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:56:21.092433" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-cgrnxm" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 10:57:56 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc000939b60>: failed waiting for hcpCluster="admin-cred-lifecycle-8bwmgr" in resourcegroup="admin-credential-lifecycle-test-8h9b62" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-8bwmgr in resource group admin-credential-lifecycle-test-8h9b62, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc000939b60>: failed waiting for hcpCluster="admin-cred-lifecycle-8bwmgr" in resourcegroup="admin-credential-lifecycle-test-8h9b62" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-8bwmgr in resource group admin-credential-lifecycle-test-8h9b62, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 10:57:56 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc001916420>: failed waiting for hcpCluster="cluster-zstream-4-19-mlfmcg" in resourcegroup="rg-zstream-upgrade-4-19-7ks759" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-mlfmcg in resource group rg-zstream-upgrade-4-19-7ks759, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc001916420>: failed waiting for hcpCluster="cluster-zstream-4-19-mlfmcg" in resourcegroup="rg-zstream-upgrade-4-19-7ks759" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-mlfmcg in resource group rg-zstream-upgrade-4-19-7ks759, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 10:57:56 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure dRequestContent\",\n \"message\": \"Invalid value: 201: must be less than or equal to 200\",\n \"target\": \"properties.replicas\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: creating a nodepool @ 04/10/26 12:35:29.989 STEP: attempting to update nodepool version to higher than cluster version @ 04/10/26 12:48:40.069 "ts"="2026-04-10 12:48:41.506501" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r9tmjd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n \"target\": \"properties.version.id\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: attempting to update immutable platform profile fields @ 04/10/26 12:48:41.506 "ts"="2026-04-10 12:48:42.314595" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r9tmjd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"4 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.vmSize\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.osDisk\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.availabilityZone\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-04-10 12:48:42.593186" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:48:42.593298" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-t7cvxj" "resourceGroup"="rg-negative-tests-r9tmjd" "ts"="2026-04-10 12:48:43.140137" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-t7cvxj" "resourceGroup"="" "ts"="2026-04-10 12:48:45.405483" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 12:48:45.405582" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 12:48:45.405650" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:48:45.407151" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-81" "ts"="2026-04-10 12:48:45.407168" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-r9tmjd" "ts"="2026-04-10 12:48:45.585306" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-81" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:48:45.685982" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-r9tmjd" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-04-10 12:48:45.901268" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-81" "deployment"="msi-deployment-81" "ts"="2026-04-10 12:48:46.030194" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-r9tmjd" "deployment"="customer-infra-negative-tests-cluster-t7cvxj" "ts"="2026-04-10 12:48:46.498826" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:48:46.498977" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:48:46.499112" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-r9tmjd" Command Error: signal: killed Deserializaion Error: no output from command dRequestContent\",\n \"message\": \"Invalid value: 201: must be less than or equal to 200\",\n \"target\": \"properties.replicas\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: creating a nodepool @ 04/10/26 12:35:29.989 STEP: attempting to update nodepool version to higher than cluster version @ 04/10/26 12:48:40.069 "ts"="2026-04-10 12:48:41.506501" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r9tmjd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n \"target\": \"properties.version.id\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: attempting to update immutable platform profile fields @ 04/10/26 12:48:41.506 "ts"="2026-04-10 12:48:42.314595" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-r9tmjd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"4 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.vmSize\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.osDisk\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.availabilityZone\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-04-10 12:48:42.593186" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:48:42.593298" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-t7cvxj" "resourceGroup"="rg-negative-tests-r9tmjd" "ts"="2026-04-10 12:48:43.140137" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-t7cvxj" "resourceGroup"="" "ts"="2026-04-10 12:48:45.405483" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 12:48:45.405582" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 12:48:45.405650" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:48:45.407151" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-81" "ts"="2026-04-10 12:48:45.407168" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-r9tmjd" "ts"="2026-04-10 12:48:45.585306" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-81" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:48:45.685982" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-r9tmjd" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-04-10 12:48:45.901268" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-81" "deployment"="msi-deployment-81" "ts"="2026-04-10 12:48:46.030194" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-r9tmjd" "deployment"="customer-infra-negative-tests-cluster-t7cvxj" "ts"="2026-04-10 12:48:46.498826" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:48:46.498977" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:48:46.499112" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-r9tmjd" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 10:57:56 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775822557 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 12:02:37.604097 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 12:02:37.689386 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 12:02:37.689749" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:880" "ts"="2026-04-10 12:02:37.700441" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 12:02:37.7 "ts"="2026-04-10 12:02:37.700555" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gntxgv-z6ddmq" STEP: creating cluster parameters at control plane version @ 04/10/26 12:02:41.704 W0410 12:02:41.803680 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 12:02:41.803 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 12:03:52.723 "ts"="2026-04-10 12:03:52.723365" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gntxgv" "resourceGroup"="rg-np-version-upgrade-gntxgv-z6ddmq" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:20:10.496 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:35:31.882 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:37:32.153 W0410 12:37:32.309157 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 12:37:32.309 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 12:37:32.515 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:38:17.362 Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775822557 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 12:02:37.604097 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 12:02:37.689386 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 12:02:37.689749" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:880" "ts"="2026-04-10 12:02:37.700441" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 12:02:37.7 "ts"="2026-04-10 12:02:37.700555" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gntxgv-z6ddmq" STEP: creating cluster parameters at control plane version @ 04/10/26 12:02:41.704 W0410 12:02:41.803680 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 12:02:41.803 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 12:03:52.723 "ts"="2026-04-10 12:03:52.723365" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gntxgv" "resourceGroup"="rg-np-version-upgrade-gntxgv-z6ddmq" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:20:10.496 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:35:31.882 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:37:32.153 W0410 12:37:32.309157 880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 12:37:32.309 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 12:37:32.515 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:38:17.362 Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 10:57:56 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure t.openshift.io/managed"="true" "key2"="value2" "kubernetes.io/os"="linux" "node-role.kubernetes.io/worker"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="westeurope" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/nodePool"="s5o9c5a9h3i2v6w-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/arch"="amd64" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="s5o9c5a9h3i2v6w-np-1-lx4bc-7qxvc" "topology.kubernetes.io/region"="westeurope"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-10 12:54:12.444384" "level"=0 "msg"="Node" "name"="s5o9c5a9h3i2v6w-np-1-lx4bc-p4tm7" "labels"={"kubernetes.io/arch"="amd64" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/managed"="true" "kubernetes.io/hostname"="s5o9c5a9h3i2v6w-np-1-lx4bc-p4tm7" "node-role.kubernetes.io/worker"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="westeurope" "failure-domain.beta.kubernetes.io/zone"="2" "kubernetes.io/os"="linux" "topology.kubernetes.io/zone"="2" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "node.openshift.io/os_id"="rhel" "hypershift.openshift.io/nodePool"="s5o9c5a9h3i2v6w-np-1"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-10 12:54:12.444422" "level"=0 "msg"="Node" "name"="s5o9c5a9h3i2v6w-np-1-lx4bc-x9fkb" "labels"={"beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="westeurope" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="s5o9c5a9h3i2v6w-np-1" "topology.kubernetes.io/zone"="0" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/hostname"="s5o9c5a9h3i2v6w-np-1-lx4bc-x9fkb" "kubernetes.io/os"="linux" "node-role.kubernetes.io/worker"="" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/region"="westeurope" "beta.kubernetes.io/arch"="amd64" "key1"="value1" "kubernetes.io/arch"="amd64" "topology.disk.csi.azure.com/zone"="" "beta.kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/10/26 12:54:12.444 STEP: verifying original taints persist on 2 nodes @ 04/10/26 12:54:12.444 "ts"="2026-04-10 12:54:12.444639" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:54:12.444722" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-wgg6nm" "resourceGroup"="rg-np-labels-taints-2qpnkd" "ts"="2026-04-10 12:54:12.819345" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-wgg6nm" "resourceGroup"="" "ts"="2026-04-10 12:54:14.809808" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 12:54:14.809931" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 12:54:14.810001" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:54:14.811654" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-79" "ts"="2026-04-10 12:54:14.811682" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-2qpnkd" "ts"="2026-04-10 12:54:14.976571" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-2qpnkd" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-10 12:54:15.157403" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-79" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:54:15.468975" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-2qpnkd" "deployment"="customer-infra-cluster-np-labels-taints-wgg6nm" "ts"="2026-04-10 12:54:15.516325" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-79" "deployment"="msi-deployment-79" "ts"="2026-04-10 12:54:15.711640" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:54:15.711769" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:54:15.711872" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-2qpnkd" Command Error: signal: killed Deserializaion Error: no output from command t.openshift.io/managed"="true" "key2"="value2" "kubernetes.io/os"="linux" "node-role.kubernetes.io/worker"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="westeurope" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/nodePool"="s5o9c5a9h3i2v6w-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/arch"="amd64" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="s5o9c5a9h3i2v6w-np-1-lx4bc-7qxvc" "topology.kubernetes.io/region"="westeurope"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-10 12:54:12.444384" "level"=0 "msg"="Node" "name"="s5o9c5a9h3i2v6w-np-1-lx4bc-p4tm7" "labels"={"kubernetes.io/arch"="amd64" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/managed"="true" "kubernetes.io/hostname"="s5o9c5a9h3i2v6w-np-1-lx4bc-p4tm7" "node-role.kubernetes.io/worker"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="westeurope" "failure-domain.beta.kubernetes.io/zone"="2" "kubernetes.io/os"="linux" "topology.kubernetes.io/zone"="2" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "node.openshift.io/os_id"="rhel" "hypershift.openshift.io/nodePool"="s5o9c5a9h3i2v6w-np-1"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-10 12:54:12.444422" "level"=0 "msg"="Node" "name"="s5o9c5a9h3i2v6w-np-1-lx4bc-x9fkb" "labels"={"beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="westeurope" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="s5o9c5a9h3i2v6w-np-1" "topology.kubernetes.io/zone"="0" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/hostname"="s5o9c5a9h3i2v6w-np-1-lx4bc-x9fkb" "kubernetes.io/os"="linux" "node-role.kubernetes.io/worker"="" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/region"="westeurope" "beta.kubernetes.io/arch"="amd64" "key1"="value1" "kubernetes.io/arch"="amd64" "topology.disk.csi.azure.com/zone"="" "beta.kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/10/26 12:54:12.444 STEP: verifying original taints persist on 2 nodes @ 04/10/26 12:54:12.444 "ts"="2026-04-10 12:54:12.444639" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:54:12.444722" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-wgg6nm" "resourceGroup"="rg-np-labels-taints-2qpnkd" "ts"="2026-04-10 12:54:12.819345" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-wgg6nm" "resourceGroup"="" "ts"="2026-04-10 12:54:14.809808" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 12:54:14.809931" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 12:54:14.810001" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:54:14.811654" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-79" "ts"="2026-04-10 12:54:14.811682" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-2qpnkd" "ts"="2026-04-10 12:54:14.976571" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-2qpnkd" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-10 12:54:15.157403" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-79" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:54:15.468975" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-2qpnkd" "deployment"="customer-infra-cluster-np-labels-taints-wgg6nm" "ts"="2026-04-10 12:54:15.516325" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-79" "deployment"="msi-deployment-79" "ts"="2026-04-10 12:54:15.711640" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:54:15.711769" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:54:15.711872" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-2qpnkd" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 10:57:56 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure 11 STEP: creating customer infrastructure and managed identities for third cluster @ 04/10/26 12:51:13.197 STEP: attempting to create a third cluster using the same managed resource group as the second cluster @ 04/10/26 12:51:52.918 "ts"="2026-04-10 12:51:52.918883" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster3" "resourceGroup"="customer-rg-l8rh58" STEP: verifying that the managed resource group still exists @ 04/10/26 12:51:54.38 STEP: testing resource group-level cluster listing @ 04/10/26 12:51:54.734 "ts"="2026-04-10 12:51:55.069173" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:51:55.069279" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-m8xtvh" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:51:55.240577" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-m8xtvh" "resourceGroup"="" "ts"="2026-04-10 12:51:57.252380" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-pg7qtn" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:51:57.630267" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-pg7qtn" "resourceGroup"="" "ts"="2026-04-10 12:51:59.685527" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster3-zvf5lf" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:51:59.858444" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster3-zvf5lf" "resourceGroup"="" "ts"="2026-04-10 12:52:01.491661" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-04-10 12:52:01.491763" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-04-10 12:52:01.491828" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:52:01.493244" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-44" "ts"="2026-04-10 12:52:01.493255" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "ts"="2026-04-10 12:52:01.493268" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:52:01.493296" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-108" "ts"="2026-04-10 12:52:01.646619" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-108" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:52:01.653898" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-44" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:52:01.685532" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:52:01.724961" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="pooledNonMsiScopedAssignments-igpwm4vimfr3y" "ts"="2026-04-10 12:52:01.776009" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-108" "deployment"="msi-deployment-108" "ts"="2026-04-10 12:52:01.783539" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-44" "deployment"="msi-deployment-44" "ts"="2026-04-10 12:52:01.867495" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="customer-infra-basic-hcp-cluster3-zvf5lf" "ts"="2026-04-10 12:52:02.032517" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-04-10 12:52:02.036082" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="msi-deployment-25" "ts"="2026-04-10 12:52:02.328126" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="customer-infra-basic-hcp-cluster2-pg7qtn" "ts"="2026-04-10 12:52:03.086364" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-04-10 12:52:03.725551" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="customer-infra-basic-hcp-cluster-m8xtvh" "ts"="2026-04-10 12:52:04.092347" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:52:04.092476" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:52:04.092575" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-l8rh58" Command Error: signal: killed Deserializaion Error: no output from command 11 STEP: creating customer infrastructure and managed identities for third cluster @ 04/10/26 12:51:13.197 STEP: attempting to create a third cluster using the same managed resource group as the second cluster @ 04/10/26 12:51:52.918 "ts"="2026-04-10 12:51:52.918883" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster3" "resourceGroup"="customer-rg-l8rh58" STEP: verifying that the managed resource group still exists @ 04/10/26 12:51:54.38 STEP: testing resource group-level cluster listing @ 04/10/26 12:51:54.734 "ts"="2026-04-10 12:51:55.069173" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 12:51:55.069279" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-m8xtvh" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:51:55.240577" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-m8xtvh" "resourceGroup"="" "ts"="2026-04-10 12:51:57.252380" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-pg7qtn" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:51:57.630267" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-pg7qtn" "resourceGroup"="" "ts"="2026-04-10 12:51:59.685527" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster3-zvf5lf" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:51:59.858444" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster3-zvf5lf" "resourceGroup"="" "ts"="2026-04-10 12:52:01.491661" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-04-10 12:52:01.491763" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-04-10 12:52:01.491828" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 12:52:01.493244" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-44" "ts"="2026-04-10 12:52:01.493255" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "ts"="2026-04-10 12:52:01.493268" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-l8rh58" "ts"="2026-04-10 12:52:01.493296" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-108" "ts"="2026-04-10 12:52:01.646619" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-108" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:52:01.653898" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-44" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:52:01.685532" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 12:52:01.724961" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="pooledNonMsiScopedAssignments-igpwm4vimfr3y" "ts"="2026-04-10 12:52:01.776009" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-108" "deployment"="msi-deployment-108" "ts"="2026-04-10 12:52:01.783539" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-44" "deployment"="msi-deployment-44" "ts"="2026-04-10 12:52:01.867495" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="customer-infra-basic-hcp-cluster3-zvf5lf" "ts"="2026-04-10 12:52:02.032517" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-04-10 12:52:02.036082" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="msi-deployment-25" "ts"="2026-04-10 12:52:02.328126" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="customer-infra-basic-hcp-cluster2-pg7qtn" "ts"="2026-04-10 12:52:03.086364" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-04-10 12:52:03.725551" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-l8rh58" "deployment"="customer-infra-basic-hcp-cluster-m8xtvh" "ts"="2026-04-10 12:52:04.092347" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 12:52:04.092476" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 12:52:04.092575" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-l8rh58" Command Error: signal: killed Deserializaion Error: no output from command ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 10:57:56 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:118]: Unexpected error: <fmt.wrapError \| 0xc000f1e3a0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-7wck67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:118]: Unexpected error: <fmt.wrapError \| 0xc000f1e3a0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-7wck67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred unknown (1) Process did not finish before 2h0m0s timeout 10:57:56 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T13:01:55Z"}
08:44:15	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
08:42:20	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	2	e2e (1) Deserializaion Error: no output from command 08:42:20 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure idRequestContent\",\n \"message\": \"Invalid value: 201: must be less than or equal to 200\",\n \"target\": \"properties.replicas\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: creating a nodepool @ 04/10/26 10:00:15.885 STEP: attempting to update nodepool version to higher than cluster version @ 04/10/26 10:30:37.398 "ts"="2026-04-10 10:30:38.320547" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zvckx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n \"target\": \"properties.version.id\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: attempting to update immutable platform profile fields @ 04/10/26 10:30:38.32 "ts"="2026-04-10 10:30:38.793824" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zvckx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"4 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.vmSize\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.osDisk\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.availabilityZone\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-04-10 10:30:38.933364" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:30:38.933477" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-8w95lb" "resourceGroup"="rg-negative-tests-zvckx9" "ts"="2026-04-10 10:30:39.052915" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-8w95lb" "resourceGroup"="" "ts"="2026-04-10 10:30:40.502197" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 10:30:40.502303" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 10:30:40.502363" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:30:40.503883" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-27" "ts"="2026-04-10 10:30:40.503901" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-zvckx9" "ts"="2026-04-10 10:30:40.649649" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-zvckx9" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-04-10 10:30:40.710044" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-zvckx9" "deployment"="customer-infra-negative-tests-cluster-8w95lb" "ts"="2026-04-10 10:30:41.332013" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-27" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:30:41.859258" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-27" "deployment"="msi-deployment-27" "ts"="2026-04-10 10:30:42.052837" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:30:42.052956" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:30:42.053071" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-zvckx9" Command Error: signal: killed Deserializaion Error: no output from command idRequestContent\",\n \"message\": \"Invalid value: 201: must be less than or equal to 200\",\n \"target\": \"properties.replicas\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: creating a nodepool @ 04/10/26 10:00:15.885 STEP: attempting to update nodepool version to higher than cluster version @ 04/10/26 10:30:37.398 "ts"="2026-04-10 10:30:38.320547" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zvckx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n \"target\": \"properties.version.id\"\n }\n}\n--------------------------------------------------------------------------------\n" STEP: attempting to update immutable platform profile fields @ 04/10/26 10:30:38.32 "ts"="2026-04-10 10:30:38.793824" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zvckx9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"4 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.vmSize\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.osDisk\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Forbidden: field is immutable\",\n \"target\": \"properties.platform.availabilityZone\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" "ts"="2026-04-10 10:30:38.933364" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:30:38.933477" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-8w95lb" "resourceGroup"="rg-negative-tests-zvckx9" "ts"="2026-04-10 10:30:39.052915" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-8w95lb" "resourceGroup"="" "ts"="2026-04-10 10:30:40.502197" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 10:30:40.502303" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz" "ts"="2026-04-10 10:30:40.502363" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:30:40.503883" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-27" "ts"="2026-04-10 10:30:40.503901" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-zvckx9" "ts"="2026-04-10 10:30:40.649649" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-zvckx9" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy" "ts"="2026-04-10 10:30:40.710044" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-zvckx9" "deployment"="customer-infra-negative-tests-cluster-8w95lb" "ts"="2026-04-10 10:30:41.332013" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-27" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:30:41.859258" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-27" "deployment"="msi-deployment-27" "ts"="2026-04-10 10:30:42.052837" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:30:42.052956" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:30:42.053071" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-zvckx9" Command Error: signal: killed Deserializaion Error: no output from command unknown (1) Process did not finish before 2h0m0s timeout 08:42:20 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T10:44:38Z"}
08:37:16	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	5	e2e (4) Deserializaion Error: no output from command 08:37:16 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure =================== Random Seed: 1775813881 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226 W0410 09:38:01.837888 894 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 09:38:01.916295 894 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 09:38:01.916585" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest\|pid:894" "ts"="2026-04-10 09:38:02.009014" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 09:38:02.009 "ts"="2026-04-10 09:38:02.009151" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" STEP: creating cluster parameters at control plane version @ 04/10/26 09:38:05.69 STEP: creating customer resources @ 04/10/26 09:38:05.788 STEP: creating the HCP cluster with version 4.20.17 @ 04/10/26 09:39:06.67 "ts"="2026-04-10 09:39:06.671088" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-xk7xd6" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 09:52:44.143 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 10:11:12.29 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 10:13:24.961 STEP: capturing node release images before upgrade @ 04/10/26 10:13:25.137 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/10/26 10:13:25.341 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 10:14:12.28 STEP: verifying node pool GET still reflects the new version @ 04/10/26 10:36:18.546 "ts"="2026-04-10 10:36:18.945641" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:36:18.945925" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-xk7xd6-mlggdr" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "ts"="2026-04-10 10:36:19.435256" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-xk7xd6-mlggdr" "resourceGroup"="" "ts"="2026-04-10 10:36:20.855485" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 10:36:20.855726" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 10:36:20.855961" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:36:20.858515" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-11" "ts"="2026-04-10 10:36:20.858534" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "ts"="2026-04-10 10:36:21.566430" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-11" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:36:21.718120" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-11" "deployment"="msi-deployment-11" "ts"="2026-04-10 10:36:21.767633" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "deployment"="pooledNonMsiScopedAssignments-wlvrodc3sbnq4" "ts"="2026-04-10 10:36:22.824528" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "deployment"="customer-infra-np-version-upgrade-cluster-xk7xd6-mlggdr" "ts"="2026-04-10 10:36:23.837140" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:36:23.837297" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:36:23.837475" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" Command Error: signal: killed Deserializaion Error: no output from command =================== Random Seed: 1775813881 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226 W0410 09:38:01.837888 894 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 09:38:01.916295 894 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 09:38:01.916585" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest\|pid:894" "ts"="2026-04-10 09:38:02.009014" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 09:38:02.009 "ts"="2026-04-10 09:38:02.009151" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" STEP: creating cluster parameters at control plane version @ 04/10/26 09:38:05.69 STEP: creating customer resources @ 04/10/26 09:38:05.788 STEP: creating the HCP cluster with version 4.20.17 @ 04/10/26 09:39:06.67 "ts"="2026-04-10 09:39:06.671088" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-xk7xd6" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 09:52:44.143 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 10:11:12.29 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 10:13:24.961 STEP: capturing node release images before upgrade @ 04/10/26 10:13:25.137 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/10/26 10:13:25.341 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 10:14:12.28 STEP: verifying node pool GET still reflects the new version @ 04/10/26 10:36:18.546 "ts"="2026-04-10 10:36:18.945641" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:36:18.945925" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-xk7xd6-mlggdr" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "ts"="2026-04-10 10:36:19.435256" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-xk7xd6-mlggdr" "resourceGroup"="" "ts"="2026-04-10 10:36:20.855485" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 10:36:20.855726" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-10 10:36:20.855961" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:36:20.858515" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-11" "ts"="2026-04-10 10:36:20.858534" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "ts"="2026-04-10 10:36:21.566430" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-11" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:36:21.718120" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-11" "deployment"="msi-deployment-11" "ts"="2026-04-10 10:36:21.767633" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "deployment"="pooledNonMsiScopedAssignments-wlvrodc3sbnq4" "ts"="2026-04-10 10:36:22.824528" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" "deployment"="customer-infra-np-version-upgrade-cluster-xk7xd6-mlggdr" "ts"="2026-04-10 10:36:23.837140" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:36:23.837297" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:36:23.837475" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-xk7xd6-t4x7hz" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 08:37:16 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775813878 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 09:37:58.491380 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 09:37:58.569259 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 09:37:58.569629" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:881" "ts"="2026-04-10 09:38:01.525392" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="3s" STEP: creating resource group @ 04/10/26 09:38:01.525 "ts"="2026-04-10 09:38:01.525521" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-b77f69-wwx7xr" STEP: creating cluster parameters at control plane version @ 04/10/26 09:38:06.084 W0410 09:38:06.171994 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 09:38:06.172 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 09:38:58.106 "ts"="2026-04-10 09:38:58.106324" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-b77f69" "resourceGroup"="rg-np-version-upgrade-b77f69-wwx7xr" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 09:53:35.552 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 10:17:50.709 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 10:20:21.895 W0410 10:20:22.019930 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 10:20:22.019 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 10:20:22.208 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 10:21:09.14 Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775813878 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 09:37:58.491380 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 09:37:58.569259 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 09:37:58.569629" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:881" "ts"="2026-04-10 09:38:01.525392" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="3s" STEP: creating resource group @ 04/10/26 09:38:01.525 "ts"="2026-04-10 09:38:01.525521" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-b77f69-wwx7xr" STEP: creating cluster parameters at control plane version @ 04/10/26 09:38:06.084 W0410 09:38:06.171994 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 09:38:06.172 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 09:38:58.106 "ts"="2026-04-10 09:38:58.106324" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-b77f69" "resourceGroup"="rg-np-version-upgrade-b77f69-wwx7xr" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 09:53:35.552 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 10:17:50.709 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 10:20:21.895 W0410 10:20:22.019930 881 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 10:20:22.019 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 10:20:22.208 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 10:21:09.14 Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 08:37:16 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure els"={"kubernetes.io/arch"="amd64" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/zone"="0" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "kubernetes.io/os"="linux" "topology.kubernetes.io/region"="westeurope" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/nodePool"="m6j3f3x0x1m5j0h-np-1" "kubernetes.io/hostname"="m6j3f3x0x1m5j0h-np-1-xw2qn-9pslf" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/arch"="amd64" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-10 10:33:52.686824" "level"=0 "msg"="Node" "name"="m6j3f3x0x1m5j0h-np-1-xw2qn-n9pqg" "labels"={"kubernetes.io/hostname"="m6j3f3x0x1m5j0h-np-1-xw2qn-n9pqg" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/managed"="true" "key2"="value2" "node-role.kubernetes.io/worker"="" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/region"="westeurope" "beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/nodePool"="m6j3f3x0x1m5j0h-np-1" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux" "kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="0" "kubernetes.io/arch"="amd64"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-10 10:33:52.686885" "level"=0 "msg"="Node" "name"="m6j3f3x0x1m5j0h-np-1-xw2qn-rkwb9" "labels"={"failure-domain.beta.kubernetes.io/zone"="1" "kubernetes.io/hostname"="m6j3f3x0x1m5j0h-np-1-xw2qn-rkwb9" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/nodePool"="m6j3f3x0x1m5j0h-np-1" "key1"="value1" "node-role.kubernetes.io/worker"="" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/os"="linux" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="westeurope" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "topology.kubernetes.io/zone"="1"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/10/26 10:33:52.686 STEP: verifying original taints persist on 2 nodes @ 04/10/26 10:33:52.686 "ts"="2026-04-10 10:33:52.687161" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:33:52.687268" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-6phmrh" "resourceGroup"="rg-np-labels-taints-c56s4p" "ts"="2026-04-10 10:33:52.888703" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-6phmrh" "resourceGroup"="" "ts"="2026-04-10 10:33:54.820943" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 10:33:54.821065" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 10:33:54.821146" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:33:54.822707" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "ts"="2026-04-10 10:33:54.822719" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-c56s4p" "ts"="2026-04-10 10:33:55.000905" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:33:55.525859" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-c56s4p" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-10 10:33:55.533515" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="msi-deployment-4" "ts"="2026-04-10 10:33:55.669144" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-c56s4p" "deployment"="customer-infra-cluster-np-labels-taints-6phmrh" "ts"="2026-04-10 10:33:56.557963" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:33:56.558106" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:33:56.558240" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-c56s4p" Command Error: signal: killed Deserializaion Error: no output from command els"={"kubernetes.io/arch"="amd64" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/zone"="0" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "kubernetes.io/os"="linux" "topology.kubernetes.io/region"="westeurope" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/nodePool"="m6j3f3x0x1m5j0h-np-1" "kubernetes.io/hostname"="m6j3f3x0x1m5j0h-np-1-xw2qn-9pslf" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/arch"="amd64" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-10 10:33:52.686824" "level"=0 "msg"="Node" "name"="m6j3f3x0x1m5j0h-np-1-xw2qn-n9pqg" "labels"={"kubernetes.io/hostname"="m6j3f3x0x1m5j0h-np-1-xw2qn-n9pqg" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/managed"="true" "key2"="value2" "node-role.kubernetes.io/worker"="" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/region"="westeurope" "beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/nodePool"="m6j3f3x0x1m5j0h-np-1" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux" "kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="0" "kubernetes.io/arch"="amd64"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] "ts"="2026-04-10 10:33:52.686885" "level"=0 "msg"="Node" "name"="m6j3f3x0x1m5j0h-np-1-xw2qn-rkwb9" "labels"={"failure-domain.beta.kubernetes.io/zone"="1" "kubernetes.io/hostname"="m6j3f3x0x1m5j0h-np-1-xw2qn-rkwb9" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/nodePool"="m6j3f3x0x1m5j0h-np-1" "key1"="value1" "node-role.kubernetes.io/worker"="" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="westeurope" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/os"="linux" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="westeurope" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "topology.kubernetes.io/zone"="1"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/10/26 10:33:52.686 STEP: verifying original taints persist on 2 nodes @ 04/10/26 10:33:52.686 "ts"="2026-04-10 10:33:52.687161" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:33:52.687268" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-6phmrh" "resourceGroup"="rg-np-labels-taints-c56s4p" "ts"="2026-04-10 10:33:52.888703" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-6phmrh" "resourceGroup"="" "ts"="2026-04-10 10:33:54.820943" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 10:33:54.821065" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-10 10:33:54.821146" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:33:54.822707" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "ts"="2026-04-10 10:33:54.822719" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-c56s4p" "ts"="2026-04-10 10:33:55.000905" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:33:55.525859" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-c56s4p" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-10 10:33:55.533515" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="msi-deployment-4" "ts"="2026-04-10 10:33:55.669144" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-c56s4p" "deployment"="customer-infra-cluster-np-labels-taints-6phmrh" "ts"="2026-04-10 10:33:56.557963" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:33:56.558106" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:33:56.558240" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-c56s4p" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 08:37:16 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure nly 0 are assigned" "attempt"=11 "retryIn"="1m0s" "elapsed"="10m0s" "ts"="2026-04-10 09:51:38.983444" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=12 "retryIn"="1m0s" "elapsed"="11m0s" "ts"="2026-04-10 09:52:38.985566" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=13 "retryIn"="1m0s" "elapsed"="12m0s" "ts"="2026-04-10 09:53:38.989067" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=14 "retryIn"="1m0s" "elapsed"="13m0s" "ts"="2026-04-10 09:54:38.991008" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=15 "retryIn"="1m0s" "elapsed"="14m0s" "ts"="2026-04-10 09:55:38.993317" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=16 "retryIn"="1m0s" "elapsed"="15m0s" "ts"="2026-04-10 09:56:39.001085" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=17 "elapsed"="16m0s" STEP: creating a resource group @ 04/10/26 09:56:39.001 "ts"="2026-04-10 09:56:39.001171" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-hfmst4" STEP: creating cluster parameters @ 04/10/26 09:56:43.2 STEP: creating customer resources @ 04/10/26 09:56:43.2 STEP: creating the HCP cluster @ 04/10/26 09:57:45.497 "ts"="2026-04-10 09:57:45.497279" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-hfmst4" STEP: creating a storage account for shoebox logs @ 04/10/26 10:09:39.477 "ts"="2026-04-10 10:10:00.273925" "level"=0 "msg"="storage account created" "name"="shoeboxm5mvvd" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-hfmst4/providers/Microsoft.Storage/storageAccounts/shoeboxm5mvvd" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 10:10:00.273 "ts"="2026-04-10 10:10:02.370642" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 10:10:02.37 "ts"="2026-04-10 10:38:02.717727" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxm5mvvd" "timestamp"="2026-04-10T10:38:02Z" "ts"="2026-04-10 10:38:02.717995" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:38:02.718099" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-cvrd95" "resourceGroup"="shoebox-cluster-hfmst4" "ts"="2026-04-10 10:38:03.225584" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-cvrd95" "resourceGroup"="" "ts"="2026-04-10 10:38:05.408173" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 10:38:05.408414" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 10:38:05.408565" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:38:05.411248" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "ts"="2026-04-10 10:38:05.411345" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-hfmst4" "ts"="2026-04-10 10:38:05.728184" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-hfmst4" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 10:38:05.767626" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:38:05.884292" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-hfmst4" "deployment"="customer-infra-shoebox-hcp-cluster-cvrd95" "ts"="2026-04-10 10:38:05.908471" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="msi-deployment-78" "ts"="2026-04-10 10:38:06.218123" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:38:06.218519" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:38:06.218829" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-hfmst4" Command Error: signal: killed Deserializaion Error: no output from command nly 0 are assigned" "attempt"=11 "retryIn"="1m0s" "elapsed"="10m0s" "ts"="2026-04-10 09:51:38.983444" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=12 "retryIn"="1m0s" "elapsed"="11m0s" "ts"="2026-04-10 09:52:38.985566" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=13 "retryIn"="1m0s" "elapsed"="12m0s" "ts"="2026-04-10 09:53:38.989067" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=14 "retryIn"="1m0s" "elapsed"="13m0s" "ts"="2026-04-10 09:54:38.991008" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=15 "retryIn"="1m0s" "elapsed"="14m0s" "ts"="2026-04-10 09:55:38.993317" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=16 "retryIn"="1m0s" "elapsed"="15m0s" "ts"="2026-04-10 09:56:39.001085" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=17 "elapsed"="16m0s" STEP: creating a resource group @ 04/10/26 09:56:39.001 "ts"="2026-04-10 09:56:39.001171" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-hfmst4" STEP: creating cluster parameters @ 04/10/26 09:56:43.2 STEP: creating customer resources @ 04/10/26 09:56:43.2 STEP: creating the HCP cluster @ 04/10/26 09:57:45.497 "ts"="2026-04-10 09:57:45.497279" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-hfmst4" STEP: creating a storage account for shoebox logs @ 04/10/26 10:09:39.477 "ts"="2026-04-10 10:10:00.273925" "level"=0 "msg"="storage account created" "name"="shoeboxm5mvvd" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-hfmst4/providers/Microsoft.Storage/storageAccounts/shoeboxm5mvvd" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 10:10:00.273 "ts"="2026-04-10 10:10:02.370642" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 10:10:02.37 "ts"="2026-04-10 10:38:02.717727" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxm5mvvd" "timestamp"="2026-04-10T10:38:02Z" "ts"="2026-04-10 10:38:02.717995" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 10:38:02.718099" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-cvrd95" "resourceGroup"="shoebox-cluster-hfmst4" "ts"="2026-04-10 10:38:03.225584" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-cvrd95" "resourceGroup"="" "ts"="2026-04-10 10:38:05.408173" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 10:38:05.408414" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 10:38:05.408565" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 10:38:05.411248" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "ts"="2026-04-10 10:38:05.411345" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-hfmst4" "ts"="2026-04-10 10:38:05.728184" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-hfmst4" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 10:38:05.767626" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 10:38:05.884292" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-hfmst4" "deployment"="customer-infra-shoebox-hcp-cluster-cvrd95" "ts"="2026-04-10 10:38:05.908471" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-78" "deployment"="msi-deployment-78" "ts"="2026-04-10 10:38:06.218123" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 10:38:06.218519" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 10:38:06.218829" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-hfmst4" Command Error: signal: killed Deserializaion Error: no output from command unknown (1) Process did not finish before 2h0m0s timeout 08:37:16 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T10:42:24Z"}
02:06:29	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	21	e2e (20) Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure "ts"="2026-04-10 03:43:20.625744" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=93 "retryIn"="1m0s" "elapsed"="1h32m0s" "ts"="2026-04-10 03:44:20.628519" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=94 "retryIn"="1m0s" "elapsed"="1h33m0s" "ts"="2026-04-10 03:45:20.631193" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=95 "retryIn"="1m0s" "elapsed"="1h34m0s" "ts"="2026-04-10 03:46:20.633847" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=96 "retryIn"="1m0s" "elapsed"="1h35m0s" "ts"="2026-04-10 03:47:20.636270" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=97 "retryIn"="1m0s" "elapsed"="1h36m0s" "ts"="2026-04-10 03:48:20.642102" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=98 "retryIn"="1m0s" "elapsed"="1h37m0s" "ts"="2026-04-10 03:49:20.645334" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=99 "retryIn"="1m0s" "elapsed"="1h38m0s" "ts"="2026-04-10 03:50:20.654693" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=100 "elapsed"="1h39m0s" STEP: creating a resource group @ 04/10/26 03:50:20.654 "ts"="2026-04-10 03:50:20.654798" "level"=0 "msg"="creating resource group" "resourceGroup"="patch-tags-xkxpmj" STEP: creating cluster parameters @ 04/10/26 03:50:24.858 STEP: creating customer resources @ 04/10/26 03:50:24.858 STEP: creating the HCP cluster @ 04/10/26 03:51:33.679 "ts"="2026-04-10 03:51:33.679728" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="patch-tags-cluster" "resourceGroup"="patch-tags-xkxpmj" STEP: getting credentials @ 04/10/26 04:07:39.558 STEP: ensuring the cluster is viable @ 04/10/26 04:09:44.551 STEP: sending a PATCH request to set a tag @ 04/10/26 04:09:45.833 STEP: verifying the tag is present in the update response body @ 04/10/26 04:10:42.758 STEP: verifying the tag is present on the cluster @ 04/10/26 04:10:42.758 "ts"="2026-04-10 04:10:43.153297" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:10:43.153395" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-tags-cluster-5bk5cn" "resourceGroup"="patch-tags-xkxpmj" "ts"="2026-04-10 04:10:43.442391" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-tags-cluster-5bk5cn" "resourceGroup"="" "ts"="2026-04-10 04:10:45.963000" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-04-10 04:10:45.963106" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-04-10 04:10:45.963168" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:10:45.964991" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-64" "ts"="2026-04-10 04:10:45.965015" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-tags-xkxpmj" "ts"="2026-04-10 04:10:46.141081" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-64" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:10:46.264710" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-64" "deployment"="msi-deployment-64" "ts"="2026-04-10 04:10:46.277682" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-xkxpmj" "deployment"="pooledNonMsiScopedAssignments-fklo75htpxzxc" "ts"="2026-04-10 04:10:46.545512" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-xkxpmj" "deployment"="customer-infra-patch-tags-cluster-5bk5cn" "ts"="2026-04-10 04:10:46.811374" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:10:46.811511" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:10:46.811625" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-tags-xkxpmj" Command Error: signal: killed Deserializaion Error: no output from command "ts"="2026-04-10 03:43:20.625744" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=93 "retryIn"="1m0s" "elapsed"="1h32m0s" "ts"="2026-04-10 03:44:20.628519" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=94 "retryIn"="1m0s" "elapsed"="1h33m0s" "ts"="2026-04-10 03:45:20.631193" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=95 "retryIn"="1m0s" "elapsed"="1h34m0s" "ts"="2026-04-10 03:46:20.633847" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=96 "retryIn"="1m0s" "elapsed"="1h35m0s" "ts"="2026-04-10 03:47:20.636270" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=97 "retryIn"="1m0s" "elapsed"="1h36m0s" "ts"="2026-04-10 03:48:20.642102" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=98 "retryIn"="1m0s" "elapsed"="1h37m0s" "ts"="2026-04-10 03:49:20.645334" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=99 "retryIn"="1m0s" "elapsed"="1h38m0s" "ts"="2026-04-10 03:50:20.654693" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=100 "elapsed"="1h39m0s" STEP: creating a resource group @ 04/10/26 03:50:20.654 "ts"="2026-04-10 03:50:20.654798" "level"=0 "msg"="creating resource group" "resourceGroup"="patch-tags-xkxpmj" STEP: creating cluster parameters @ 04/10/26 03:50:24.858 STEP: creating customer resources @ 04/10/26 03:50:24.858 STEP: creating the HCP cluster @ 04/10/26 03:51:33.679 "ts"="2026-04-10 03:51:33.679728" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="patch-tags-cluster" "resourceGroup"="patch-tags-xkxpmj" STEP: getting credentials @ 04/10/26 04:07:39.558 STEP: ensuring the cluster is viable @ 04/10/26 04:09:44.551 STEP: sending a PATCH request to set a tag @ 04/10/26 04:09:45.833 STEP: verifying the tag is present in the update response body @ 04/10/26 04:10:42.758 STEP: verifying the tag is present on the cluster @ 04/10/26 04:10:42.758 "ts"="2026-04-10 04:10:43.153297" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:10:43.153395" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-patch-tags-cluster-5bk5cn" "resourceGroup"="patch-tags-xkxpmj" "ts"="2026-04-10 04:10:43.442391" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-patch-tags-cluster-5bk5cn" "resourceGroup"="" "ts"="2026-04-10 04:10:45.963000" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-04-10 04:10:45.963106" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4117c8518f426e648f352d29f774d8baa91734601e145475be95e6bea8053dce.yaml.gz" "ts"="2026-04-10 04:10:45.963168" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:10:45.964991" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-64" "ts"="2026-04-10 04:10:45.965015" "level"=0 "msg"="collecting deployments" "resourceGroup"="patch-tags-xkxpmj" "ts"="2026-04-10 04:10:46.141081" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-64" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:10:46.264710" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-64" "deployment"="msi-deployment-64" "ts"="2026-04-10 04:10:46.277682" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-xkxpmj" "deployment"="pooledNonMsiScopedAssignments-fklo75htpxzxc" "ts"="2026-04-10 04:10:46.545512" "level"=0 "msg"="collecting operations" "resourceGroup"="patch-tags-xkxpmj" "deployment"="customer-infra-patch-tags-cluster-5bk5cn" "ts"="2026-04-10 04:10:46.811374" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:10:46.811511" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:10:46.811625" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="patch-tags-xkxpmj" Command Error: signal: killed Deserializaion Error: no output from command ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 02:06:29 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapError \| 0xc00087a680>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9w9nvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:101]: Unexpected error: <fmt.wrapError \| 0xc00087a680>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9w9nvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 02:06:29 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error: <fmt.wrapError \| 0xc000a8a220>: failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-fhjd2t" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-fhjd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error: <fmt.wrapError \| 0xc000a8a220>: failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-fhjd2t" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-fhjd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- ... occurred Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure ee identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=37 "retryIn"="1m0s" "elapsed"="36m0s" "ts"="2026-04-10 03:44:28.728317" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=38 "retryIn"="1m0s" "elapsed"="37m0s" "ts"="2026-04-10 03:45:28.731638" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=39 "retryIn"="1m0s" "elapsed"="38m0s" "ts"="2026-04-10 03:46:28.734214" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=40 "retryIn"="1m0s" "elapsed"="39m0s" "ts"="2026-04-10 03:47:28.737026" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=41 "retryIn"="1m0s" "elapsed"="40m0s" "ts"="2026-04-10 03:48:28.739145" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=42 "retryIn"="1m0s" "elapsed"="41m0s" "ts"="2026-04-10 03:49:28.741381" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=43 "retryIn"="1m0s" "elapsed"="42m0s" "ts"="2026-04-10 03:50:28.743535" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=44 "retryIn"="1m0s" "elapsed"="43m0s" "ts"="2026-04-10 03:51:28.746782" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=45 "retryIn"="1m0s" "elapsed"="44m0s" "ts"="2026-04-10 03:52:28.749783" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=46 "retryIn"="1m0s" "elapsed"="45m0s" "ts"="2026-04-10 03:53:28.753253" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=47 "retryIn"="1m0s" "elapsed"="46m0s" "ts"="2026-04-10 03:54:28.755799" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=48 "retryIn"="1m0s" "elapsed"="47m0s" "ts"="2026-04-10 03:55:28.759054" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=49 "retryIn"="1m0s" "elapsed"="48m0s" "ts"="2026-04-10 03:56:28.760633" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=50 "retryIn"="1m0s" "elapsed"="49m0s" "ts"="2026-04-10 03:57:28.763733" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=51 "retryIn"="1m0s" "elapsed"="50m0s" "ts"="2026-04-10 03:58:28.766632" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=52 "retryIn"="1m0s" "elapsed"="51m0s" "ts"="2026-04-10 03:59:28.769759" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=53 "retryIn"="1m0s" "elapsed"="52m0s" "ts"="2026-04-10 04:00:28.782533" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=54 "elapsed"="53m0s" STEP: creating resource group for the HCP cluster @ 04/10/26 04:00:28.782 "ts"="2026-04-10 04:00:28.782639" "level"=0 "msg"="creating resource group" "resourceGroup"="ea-list-tsxbpj" STEP: creating cluster parameters @ 04/10/26 04:00:32.482 STEP: creating customer resources @ 04/10/26 04:00:32.482 STEP: creating HCP cluster @ 04/10/26 04:01:54.359 "ts"="2026-04-10 04:01:54.359988" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ea-listkw785z" "resourceGroup"="ea-list-tsxbpj" Command Error: signal: killed Deserializaion Error: no output from command ee identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=37 "retryIn"="1m0s" "elapsed"="36m0s" "ts"="2026-04-10 03:44:28.728317" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=38 "retryIn"="1m0s" "elapsed"="37m0s" "ts"="2026-04-10 03:45:28.731638" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=39 "retryIn"="1m0s" "elapsed"="38m0s" "ts"="2026-04-10 03:46:28.734214" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=40 "retryIn"="1m0s" "elapsed"="39m0s" "ts"="2026-04-10 03:47:28.737026" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=41 "retryIn"="1m0s" "elapsed"="40m0s" "ts"="2026-04-10 03:48:28.739145" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=42 "retryIn"="1m0s" "elapsed"="41m0s" "ts"="2026-04-10 03:49:28.741381" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=43 "retryIn"="1m0s" "elapsed"="42m0s" "ts"="2026-04-10 03:50:28.743535" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=44 "retryIn"="1m0s" "elapsed"="43m0s" "ts"="2026-04-10 03:51:28.746782" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=45 "retryIn"="1m0s" "elapsed"="44m0s" "ts"="2026-04-10 03:52:28.749783" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=46 "retryIn"="1m0s" "elapsed"="45m0s" "ts"="2026-04-10 03:53:28.753253" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=47 "retryIn"="1m0s" "elapsed"="46m0s" "ts"="2026-04-10 03:54:28.755799" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=48 "retryIn"="1m0s" "elapsed"="47m0s" "ts"="2026-04-10 03:55:28.759054" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=49 "retryIn"="1m0s" "elapsed"="48m0s" "ts"="2026-04-10 03:56:28.760633" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=50 "retryIn"="1m0s" "elapsed"="49m0s" "ts"="2026-04-10 03:57:28.763733" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=51 "retryIn"="1m0s" "elapsed"="50m0s" "ts"="2026-04-10 03:58:28.766632" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=52 "retryIn"="1m0s" "elapsed"="51m0s" "ts"="2026-04-10 03:59:28.769759" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=53 "retryIn"="1m0s" "elapsed"="52m0s" "ts"="2026-04-10 04:00:28.782533" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=54 "elapsed"="53m0s" STEP: creating resource group for the HCP cluster @ 04/10/26 04:00:28.782 "ts"="2026-04-10 04:00:28.782639" "level"=0 "msg"="creating resource group" "resourceGroup"="ea-list-tsxbpj" STEP: creating cluster parameters @ 04/10/26 04:00:32.482 STEP: creating customer resources @ 04/10/26 04:00:32.482 STEP: creating HCP cluster @ 04/10/26 04:01:54.359 "ts"="2026-04-10 04:01:54.359988" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ea-listkw785z" "resourceGroup"="ea-list-tsxbpj" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775791117 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 03:18:37.270340 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 03:18:37.350291 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 03:18:37.350682" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:875" "ts"="2026-04-10 03:18:37.359639" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 03:18:37.359 "ts"="2026-04-10 03:18:37.359757" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-qs7b4k-lv9m85" STEP: creating cluster parameters at control plane version @ 04/10/26 03:18:41.298 W0410 03:18:41.389740 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 03:18:41.389 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 03:19:52.126 "ts"="2026-04-10 03:19:52.126471" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-qs7b4k" "resourceGroup"="rg-np-version-upgrade-qs7b4k-lv9m85" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 03:36:40.072 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 03:46:19.01 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 03:48:21.962 W0410 03:48:22.094001 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 03:48:22.094 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 03:48:22.512 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 03:49:07.595 Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775791117 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:223 W0410 03:18:37.270340 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" W0410 03:18:37.350291 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" "ts"="2026-04-10 03:18:37.350682" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest\|pid:875" "ts"="2026-04-10 03:18:37.359639" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating resource group @ 04/10/26 03:18:37.359 "ts"="2026-04-10 03:18:37.359757" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-qs7b4k-lv9m85" STEP: creating cluster parameters at control plane version @ 04/10/26 03:18:41.298 W0410 03:18:41.389740 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: creating customer resources @ 04/10/26 03:18:41.389 STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 03:19:52.126 "ts"="2026-04-10 03:19:52.126471" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-qs7b4k" "resourceGroup"="rg-np-version-upgrade-qs7b4k-lv9m85" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 03:36:40.072 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 03:46:19.01 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 03:48:21.962 W0410 03:48:22.094001 875 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL" STEP: capturing node release images before upgrade @ 04/10/26 03:48:22.094 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 03:48:22.512 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 03:49:07.595 Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel Show raw failure "ts"="2026-04-10 03:37:20.814830" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=31 "retryIn"="1m0s" "elapsed"="30m0s" "ts"="2026-04-10 03:38:20.817653" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=32 "retryIn"="1m0s" "elapsed"="31m0s" "ts"="2026-04-10 03:39:20.820524" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=33 "retryIn"="1m0s" "elapsed"="32m0s" "ts"="2026-04-10 03:40:20.822516" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=34 "retryIn"="1m0s" "elapsed"="33m0s" "ts"="2026-04-10 03:41:20.825054" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=35 "retryIn"="1m0s" "elapsed"="34m0s" "ts"="2026-04-10 03:42:20.827784" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=36 "retryIn"="1m0s" "elapsed"="35m0s" "ts"="2026-04-10 03:43:20.831953" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=37 "retryIn"="1m0s" "elapsed"="36m0s" "ts"="2026-04-10 03:44:20.834246" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=38 "retryIn"="1m0s" "elapsed"="37m0s" "ts"="2026-04-10 03:45:20.838247" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=39 "retryIn"="1m0s" "elapsed"="38m0s" "ts"="2026-04-10 03:46:20.840670" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=40 "retryIn"="1m0s" "elapsed"="39m0s" "ts"="2026-04-10 03:47:20.843513" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=41 "retryIn"="1m0s" "elapsed"="40m0s" "ts"="2026-04-10 03:48:20.845948" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=42 "retryIn"="1m0s" "elapsed"="41m0s" "ts"="2026-04-10 03:49:20.848662" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=43 "retryIn"="1m0s" "elapsed"="42m0s" "ts"="2026-04-10 03:50:20.851282" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=44 "retryIn"="1m0s" "elapsed"="43m0s" "ts"="2026-04-10 03:51:20.854312" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=45 "retryIn"="1m0s" "elapsed"="44m0s" "ts"="2026-04-10 03:52:20.857003" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=46 "retryIn"="1m0s" "elapsed"="45m0s" "ts"="2026-04-10 03:53:20.889041" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=47 "elapsed"="46m0s" STEP: creating resource group @ 04/10/26 03:53:20.889 "ts"="2026-04-10 03:53:20.889167" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-cp-ystream-upgrade-4-21-q6jzn7" STEP: creating cluster parameters at install (previous minor) version @ 04/10/26 03:53:24.652 STEP: creating customer resources @ 04/10/26 03:53:24.652 STEP: creating the HCP cluster with install version 4.20.17 (previous minor 4.20.0) @ 04/10/26 03:54:45.281 "ts"="2026-04-10 03:54:45.282075" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cp-ystream-upgrade-4-21-9vfdjd" "resourceGroup"="rg-cp-ystream-upgrade-4-21-q6jzn7" Command Error: signal: killed Deserializaion Error: no output from command "ts"="2026-04-10 03:37:20.814830" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=31 "retryIn"="1m0s" "elapsed"="30m0s" "ts"="2026-04-10 03:38:20.817653" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=32 "retryIn"="1m0s" "elapsed"="31m0s" "ts"="2026-04-10 03:39:20.820524" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=33 "retryIn"="1m0s" "elapsed"="32m0s" "ts"="2026-04-10 03:40:20.822516" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=34 "retryIn"="1m0s" "elapsed"="33m0s" "ts"="2026-04-10 03:41:20.825054" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=35 "retryIn"="1m0s" "elapsed"="34m0s" "ts"="2026-04-10 03:42:20.827784" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=36 "retryIn"="1m0s" "elapsed"="35m0s" "ts"="2026-04-10 03:43:20.831953" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=37 "retryIn"="1m0s" "elapsed"="36m0s" "ts"="2026-04-10 03:44:20.834246" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=38 "retryIn"="1m0s" "elapsed"="37m0s" "ts"="2026-04-10 03:45:20.838247" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=39 "retryIn"="1m0s" "elapsed"="38m0s" "ts"="2026-04-10 03:46:20.840670" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=40 "retryIn"="1m0s" "elapsed"="39m0s" "ts"="2026-04-10 03:47:20.843513" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=41 "retryIn"="1m0s" "elapsed"="40m0s" "ts"="2026-04-10 03:48:20.845948" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=42 "retryIn"="1m0s" "elapsed"="41m0s" "ts"="2026-04-10 03:49:20.848662" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=43 "retryIn"="1m0s" "elapsed"="42m0s" "ts"="2026-04-10 03:50:20.851282" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=44 "retryIn"="1m0s" "elapsed"="43m0s" "ts"="2026-04-10 03:51:20.854312" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=45 "retryIn"="1m0s" "elapsed"="44m0s" "ts"="2026-04-10 03:52:20.857003" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=46 "retryIn"="1m0s" "elapsed"="45m0s" "ts"="2026-04-10 03:53:20.889041" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=47 "elapsed"="46m0s" STEP: creating resource group @ 04/10/26 03:53:20.889 "ts"="2026-04-10 03:53:20.889167" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-cp-ystream-upgrade-4-21-q6jzn7" STEP: creating cluster parameters at install (previous minor) version @ 04/10/26 03:53:24.652 STEP: creating customer resources @ 04/10/26 03:53:24.652 STEP: creating the HCP cluster with install version 4.20.17 (previous minor 4.20.0) @ 04/10/26 03:54:45.281 "ts"="2026-04-10 03:54:45.282075" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cp-ystream-upgrade-4-21-9vfdjd" "resourceGroup"="rg-cp-ystream-upgrade-4-21-q6jzn7" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775791077 - will randomize all specs Will run 1 of 1 specs ------------------------------ Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:67 "ts"="2026-04-10 03:17:57.722076" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Nodepool-Ephemeral-OS-Disk-should-create-a-nodepool-with-ephemeral-OS-disk-when-autoRepair-is-enabled\|pid:850" "ts"="2026-04-10 03:17:57.728052" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 03:17:57.728 "ts"="2026-04-10 03:17:57.728100" "level"=0 "msg"="creating resource group" "resourceGroup"="ephemeral-osdisk-6n9k4m" STEP: creating cluster parameters @ 04/10/26 03:18:01.893 STEP: creating customer resources (infrastructure and managed identities) @ 04/10/26 03:18:01.893 STEP: creating the HCP cluster @ 04/10/26 03:19:19.787 "ts"="2026-04-10 03:19:19.788115" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ephemeral-disk" "resourceGroup"="ephemeral-osdisk-6n9k4m" [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112 @ 04/10/26 04:04:19.788 "ts"="2026-04-10 04:04:19.789009" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:04:19.789143" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ephemeral-disk-6b7cpt" "resourceGroup"="ephemeral-osdisk-6n9k4m" "ts"="2026-04-10 04:04:20.070933" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ephemeral-disk-6b7cpt" "resourceGroup"="" "ts"="2026-04-10 04:04:22.168165" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz" "ts"="2026-04-10 04:04:22.168254" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz" "ts"="2026-04-10 04:04:22.168310" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:04:22.169635" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "ts"="2026-04-10 04:04:22.169648" "level"=0 "msg"="collecting deployments" "resourceGroup"="ephemeral-osdisk-6n9k4m" "ts"="2026-04-10 04:04:22.464446" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-6n9k4m" "deployment"="pooledNonMsiScopedAssignments-6xts4ycasve7q" "ts"="2026-04-10 04:04:22.680743" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:04:22.744099" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-6n9k4m" "deployment"="customer-infra-ephemeral-disk-6b7cpt" "ts"="2026-04-10 04:04:22.827305" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="msi-deployment-53" "ts"="2026-04-10 04:04:23.041143" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:04:23.041245" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:04:23.041354" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ephemeral-osdisk-6n9k4m" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775791077 - will randomize all specs Will run 1 of 1 specs ------------------------------ Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:67 "ts"="2026-04-10 03:17:57.722076" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Nodepool-Ephemeral-OS-Disk-should-create-a-nodepool-with-ephemeral-OS-disk-when-autoRepair-is-enabled\|pid:850" "ts"="2026-04-10 03:17:57.728052" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 03:17:57.728 "ts"="2026-04-10 03:17:57.728100" "level"=0 "msg"="creating resource group" "resourceGroup"="ephemeral-osdisk-6n9k4m" STEP: creating cluster parameters @ 04/10/26 03:18:01.893 STEP: creating customer resources (infrastructure and managed identities) @ 04/10/26 03:18:01.893 STEP: creating the HCP cluster @ 04/10/26 03:19:19.787 "ts"="2026-04-10 03:19:19.788115" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="ephemeral-disk" "resourceGroup"="ephemeral-osdisk-6n9k4m" [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112 @ 04/10/26 04:04:19.788 "ts"="2026-04-10 04:04:19.789009" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:04:19.789143" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-ephemeral-disk-6b7cpt" "resourceGroup"="ephemeral-osdisk-6n9k4m" "ts"="2026-04-10 04:04:20.070933" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-ephemeral-disk-6b7cpt" "resourceGroup"="" "ts"="2026-04-10 04:04:22.168165" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz" "ts"="2026-04-10 04:04:22.168254" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-a54a7b95d75fb7a5c86e976d7b6a232359fe80ddc62a98537cfa2cfbd9f24bac.yaml.gz" "ts"="2026-04-10 04:04:22.168310" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:04:22.169635" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "ts"="2026-04-10 04:04:22.169648" "level"=0 "msg"="collecting deployments" "resourceGroup"="ephemeral-osdisk-6n9k4m" "ts"="2026-04-10 04:04:22.464446" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-6n9k4m" "deployment"="pooledNonMsiScopedAssignments-6xts4ycasve7q" "ts"="2026-04-10 04:04:22.680743" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:04:22.744099" "level"=0 "msg"="collecting operations" "resourceGroup"="ephemeral-osdisk-6n9k4m" "deployment"="customer-infra-ephemeral-disk-6b7cpt" "ts"="2026-04-10 04:04:22.827305" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-53" "deployment"="msi-deployment-53" "ts"="2026-04-10 04:04:23.041143" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:04:23.041245" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:04:23.041354" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="ephemeral-osdisk-6n9k4m" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 02:06:29 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc0009893e0>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-rk2pg2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-rk2pg2, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc0009893e0>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-rk2pg2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-rk2pg2, error: context deadline exceeded ... occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 02:06:29 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc000f2c840>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qczcv4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-qczcv4, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc000f2c840>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qczcv4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-qczcv4, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure ="1m0s" "elapsed"="1h20m0s" "ts"="2026-04-10 03:32:20.590832" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=82 "retryIn"="1m0s" "elapsed"="1h21m0s" "ts"="2026-04-10 03:33:20.593869" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=83 "retryIn"="1m0s" "elapsed"="1h22m0s" "ts"="2026-04-10 03:34:20.597155" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=84 "retryIn"="1m0s" "elapsed"="1h23m0s" "ts"="2026-04-10 03:35:20.599426" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=85 "retryIn"="1m0s" "elapsed"="1h24m0s" "ts"="2026-04-10 03:36:20.602757" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=86 "retryIn"="1m0s" "elapsed"="1h25m0s" "ts"="2026-04-10 03:37:20.604536" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=87 "retryIn"="1m0s" "elapsed"="1h26m0s" "ts"="2026-04-10 03:38:20.606971" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=88 "retryIn"="1m0s" "elapsed"="1h27m0s" "ts"="2026-04-10 03:39:20.609391" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=89 "retryIn"="1m0s" "elapsed"="1h28m0s" "ts"="2026-04-10 03:40:20.612976" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=90 "retryIn"="1m0s" "elapsed"="1h29m0s" "ts"="2026-04-10 03:41:20.615650" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=91 "retryIn"="1m0s" "elapsed"="1h30m0s" "ts"="2026-04-10 03:42:20.618547" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=92 "retryIn"="1m0s" "elapsed"="1h31m0s" "ts"="2026-04-10 03:43:20.621167" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=93 "retryIn"="1m0s" "elapsed"="1h32m0s" "ts"="2026-04-10 03:44:20.624472" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=94 "retryIn"="1m0s" "elapsed"="1h33m0s" "ts"="2026-04-10 03:45:20.626601" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=95 "retryIn"="1m0s" "elapsed"="1h34m0s" "ts"="2026-04-10 03:46:20.629740" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=96 "retryIn"="1m0s" "elapsed"="1h35m0s" "ts"="2026-04-10 03:47:20.631439" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=97 "retryIn"="1m0s" "elapsed"="1h36m0s" "ts"="2026-04-10 03:48:20.640252" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=98 "elapsed"="1h37m0s" STEP: creating a resource group @ 04/10/26 03:48:20.64 "ts"="2026-04-10 03:48:20.640351" "level"=0 "msg"="creating resource group" "resourceGroup"="private-keyvault-5rpxss" STEP: creating cluster parameters @ 04/10/26 03:48:24.624 STEP: creating customer resources (infrastructure and managed identities) @ 04/10/26 03:48:24.624 STEP: creating the HCP cluster @ 04/10/26 03:52:10.448 "ts"="2026-04-10 03:52:10.449153" "level"=0 "msg"="Starting HCP cluster creation (v20251223preview)" "clusterName"="private-kv-cluster" "resourceGroup"="private-keyvault-5rpxss" Command Error: signal: killed Deserializaion Error: no output from command ="1m0s" "elapsed"="1h20m0s" "ts"="2026-04-10 03:32:20.590832" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=82 "retryIn"="1m0s" "elapsed"="1h21m0s" "ts"="2026-04-10 03:33:20.593869" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=83 "retryIn"="1m0s" "elapsed"="1h22m0s" "ts"="2026-04-10 03:34:20.597155" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=84 "retryIn"="1m0s" "elapsed"="1h23m0s" "ts"="2026-04-10 03:35:20.599426" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=85 "retryIn"="1m0s" "elapsed"="1h24m0s" "ts"="2026-04-10 03:36:20.602757" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=86 "retryIn"="1m0s" "elapsed"="1h25m0s" "ts"="2026-04-10 03:37:20.604536" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=87 "retryIn"="1m0s" "elapsed"="1h26m0s" "ts"="2026-04-10 03:38:20.606971" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=88 "retryIn"="1m0s" "elapsed"="1h27m0s" "ts"="2026-04-10 03:39:20.609391" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=89 "retryIn"="1m0s" "elapsed"="1h28m0s" "ts"="2026-04-10 03:40:20.612976" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=90 "retryIn"="1m0s" "elapsed"="1h29m0s" "ts"="2026-04-10 03:41:20.615650" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=91 "retryIn"="1m0s" "elapsed"="1h30m0s" "ts"="2026-04-10 03:42:20.618547" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=92 "retryIn"="1m0s" "elapsed"="1h31m0s" "ts"="2026-04-10 03:43:20.621167" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=93 "retryIn"="1m0s" "elapsed"="1h32m0s" "ts"="2026-04-10 03:44:20.624472" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=94 "retryIn"="1m0s" "elapsed"="1h33m0s" "ts"="2026-04-10 03:45:20.626601" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=95 "retryIn"="1m0s" "elapsed"="1h34m0s" "ts"="2026-04-10 03:46:20.629740" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=96 "retryIn"="1m0s" "elapsed"="1h35m0s" "ts"="2026-04-10 03:47:20.631439" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=97 "retryIn"="1m0s" "elapsed"="1h36m0s" "ts"="2026-04-10 03:48:20.640252" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=98 "elapsed"="1h37m0s" STEP: creating a resource group @ 04/10/26 03:48:20.64 "ts"="2026-04-10 03:48:20.640351" "level"=0 "msg"="creating resource group" "resourceGroup"="private-keyvault-5rpxss" STEP: creating cluster parameters @ 04/10/26 03:48:24.624 STEP: creating customer resources (infrastructure and managed identities) @ 04/10/26 03:48:24.624 STEP: creating the HCP cluster @ 04/10/26 03:52:10.448 "ts"="2026-04-10 03:52:10.449153" "level"=0 "msg"="Starting HCP cluster creation (v20251223preview)" "clusterName"="private-kv-cluster" "resourceGroup"="private-keyvault-5rpxss" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure "="1m0s" "elapsed"="1h23m0s" "ts"="2026-04-10 03:35:20.604098" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=85 "retryIn"="1m0s" "elapsed"="1h24m0s" "ts"="2026-04-10 03:36:20.607920" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=86 "retryIn"="1m0s" "elapsed"="1h25m0s" "ts"="2026-04-10 03:37:20.609946" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=87 "retryIn"="1m0s" "elapsed"="1h26m0s" "ts"="2026-04-10 03:38:20.612631" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=88 "retryIn"="1m0s" "elapsed"="1h27m0s" "ts"="2026-04-10 03:39:20.614440" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=89 "retryIn"="1m0s" "elapsed"="1h28m0s" "ts"="2026-04-10 03:40:20.618754" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=90 "retryIn"="1m0s" "elapsed"="1h29m0s" "ts"="2026-04-10 03:41:20.621295" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=91 "retryIn"="1m0s" "elapsed"="1h30m0s" "ts"="2026-04-10 03:42:20.624572" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=92 "retryIn"="1m0s" "elapsed"="1h31m0s" "ts"="2026-04-10 03:43:20.627454" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=93 "retryIn"="1m0s" "elapsed"="1h32m0s" "ts"="2026-04-10 03:44:20.629732" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=94 "retryIn"="1m0s" "elapsed"="1h33m0s" "ts"="2026-04-10 03:45:20.632735" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=95 "retryIn"="1m0s" "elapsed"="1h34m0s" "ts"="2026-04-10 03:46:20.635585" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=96 "retryIn"="1m0s" "elapsed"="1h35m0s" "ts"="2026-04-10 03:47:20.637668" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=97 "retryIn"="1m0s" "elapsed"="1h36m0s" "ts"="2026-04-10 03:48:20.643845" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=98 "retryIn"="1m0s" "elapsed"="1h37m0s" "ts"="2026-04-10 03:49:20.647679" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=99 "retryIn"="1m0s" "elapsed"="1h38m0s" "ts"="2026-04-10 03:50:20.656164" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=100 "retryIn"="1m0s" "elapsed"="1h39m0s" "ts"="2026-04-10 03:51:20.664785" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=101 "elapsed"="1h40m1s" STEP: creating a resource group @ 04/10/26 03:51:20.664 "ts"="2026-04-10 03:51:20.664925" "level"=0 "msg"="creating resource group" "resourceGroup"="pullsecret-test-2gn9mp" STEP: creating cluster parameters @ 04/10/26 03:51:24.256 STEP: creating customer resources @ 04/10/26 03:51:24.256 STEP: Creating the cluster @ 04/10/26 03:52:35.242 "ts"="2026-04-10 03:52:35.242550" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="pullsecret-hcp-cluster" "resourceGroup"="pullsecret-test-2gn9mp" STEP: Creating the node pool @ 04/10/26 04:10:42.268 Command Error: signal: killed Deserializaion Error: no output from command "="1m0s" "elapsed"="1h23m0s" "ts"="2026-04-10 03:35:20.604098" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=85 "retryIn"="1m0s" "elapsed"="1h24m0s" "ts"="2026-04-10 03:36:20.607920" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=86 "retryIn"="1m0s" "elapsed"="1h25m0s" "ts"="2026-04-10 03:37:20.609946" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=87 "retryIn"="1m0s" "elapsed"="1h26m0s" "ts"="2026-04-10 03:38:20.612631" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=88 "retryIn"="1m0s" "elapsed"="1h27m0s" "ts"="2026-04-10 03:39:20.614440" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=89 "retryIn"="1m0s" "elapsed"="1h28m0s" "ts"="2026-04-10 03:40:20.618754" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=90 "retryIn"="1m0s" "elapsed"="1h29m0s" "ts"="2026-04-10 03:41:20.621295" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=91 "retryIn"="1m0s" "elapsed"="1h30m0s" "ts"="2026-04-10 03:42:20.624572" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=92 "retryIn"="1m0s" "elapsed"="1h31m0s" "ts"="2026-04-10 03:43:20.627454" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=93 "retryIn"="1m0s" "elapsed"="1h32m0s" "ts"="2026-04-10 03:44:20.629732" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=94 "retryIn"="1m0s" "elapsed"="1h33m0s" "ts"="2026-04-10 03:45:20.632735" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=95 "retryIn"="1m0s" "elapsed"="1h34m0s" "ts"="2026-04-10 03:46:20.635585" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=96 "retryIn"="1m0s" "elapsed"="1h35m0s" "ts"="2026-04-10 03:47:20.637668" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=97 "retryIn"="1m0s" "elapsed"="1h36m0s" "ts"="2026-04-10 03:48:20.643845" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=98 "retryIn"="1m0s" "elapsed"="1h37m0s" "ts"="2026-04-10 03:49:20.647679" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=99 "retryIn"="1m0s" "elapsed"="1h38m0s" "ts"="2026-04-10 03:50:20.656164" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 1 identity containers but only 0 are assigned" "attempt"=100 "retryIn"="1m0s" "elapsed"="1h39m0s" "ts"="2026-04-10 03:51:20.664785" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=101 "elapsed"="1h40m1s" STEP: creating a resource group @ 04/10/26 03:51:20.664 "ts"="2026-04-10 03:51:20.664925" "level"=0 "msg"="creating resource group" "resourceGroup"="pullsecret-test-2gn9mp" STEP: creating cluster parameters @ 04/10/26 03:51:24.256 STEP: creating customer resources @ 04/10/26 03:51:24.256 STEP: Creating the cluster @ 04/10/26 03:52:35.242 "ts"="2026-04-10 03:52:35.242550" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="pullsecret-hcp-cluster" "resourceGroup"="pullsecret-test-2gn9mp" STEP: Creating the node pool @ 04/10/26 04:10:42.268 Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 02:06:29 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc001154720>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-f86lp7, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-f86lp7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-f86lp7, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc001154720>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-f86lp7, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-f86lp7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-f86lp7, error: context deadline exceeded ... occurred Cluster should become ready within 45 minutes 02:06:29 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s. Cluster should become ready within 45 minutes Expected <bool>: ... to be true fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s. Cluster should become ready within 45 minutes Expected <bool>: ... to be true Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775790918 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:39 "ts"="2026-04-10 03:15:18.143083" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-create-a-cluster-with-default-autoscaling-and-a-nodepool-with-autoscaling-enabled-up-to-replica-limits\|pid:823" "ts"="2026-04-10 03:15:18.149238" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: checking if the region supports availability zones @ 04/10/26 03:15:18.149 STEP: creating a resource group @ 04/10/26 03:15:19.661 "ts"="2026-04-10 03:15:19.661117" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-6x7zd5" STEP: creating cluster parameters @ 04/10/26 03:15:23.102 STEP: creating customer resources @ 04/10/26 03:15:23.102 STEP: creating the HCP cluster @ 04/10/26 03:16:46.624 "ts"="2026-04-10 03:16:46.624184" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-6x7zd5" STEP: verifying the cluster has default autoscaling parameters @ 04/10/26 03:27:40.525 STEP: getting admin credentials for the cluster @ 04/10/26 03:27:40.857 STEP: creating the AZ nodepool with 500 max replicas @ 04/10/26 03:36:33.03 STEP: verifying the AZ nodepool has the correct autoscaling configuration @ 04/10/26 03:46:22.114 STEP: verifying node count after AZ nodepool creation @ 04/10/26 03:46:30.42 STEP: updating the AZ nodepool max replicas from 500 to 4 before creating the next nodepool @ 04/10/26 03:46:31.463 STEP: verifying the AZ nodepool max replicas was updated to 4 @ 04/10/26 03:52:03.612 STEP: creating the no-AZ nodepool with 200 max replicas @ 04/10/26 03:52:03.976 STEP: verifying the no-AZ nodepool has the correct autoscaling configuration @ 04/10/26 04:01:52.182 STEP: verifying node count after no-AZ nodepool creation @ 04/10/26 04:01:53.049 "ts"="2026-04-10 04:01:54.362264" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:01:54.362332" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-pgwxc7" "resourceGroup"="np-autoscaling-6x7zd5" "ts"="2026-04-10 04:01:54.650082" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-pgwxc7" "resourceGroup"="" "ts"="2026-04-10 04:01:56.684085" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4c275018436d67b080599c8133799f947e5f8f174147f1d7309198de43b9be47.yaml.gz" "ts"="2026-04-10 04:01:56.684188" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4c275018436d67b080599c8133799f947e5f8f174147f1d7309198de43b9be47.yaml.gz" "ts"="2026-04-10 04:01:56.684258" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:01:56.685776" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-114" "ts"="2026-04-10 04:01:56.685793" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-6x7zd5" "ts"="2026-04-10 04:01:56.965161" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-6x7zd5" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs" "ts"="2026-04-10 04:01:57.199959" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-114" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:01:57.230486" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-6x7zd5" "deployment"="customer-infra-np-autoscale-cluster-pgwxc7" "ts"="2026-04-10 04:01:57.585359" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-114" "deployment"="msi-deployment-114" "ts"="2026-04-10 04:01:57.751304" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:01:57.751435" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:01:57.751569" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-6x7zd5" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775790918 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:39 "ts"="2026-04-10 03:15:18.143083" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-create-a-cluster-with-default-autoscaling-and-a-nodepool-with-autoscaling-enabled-up-to-replica-limits\|pid:823" "ts"="2026-04-10 03:15:18.149238" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: checking if the region supports availability zones @ 04/10/26 03:15:18.149 STEP: creating a resource group @ 04/10/26 03:15:19.661 "ts"="2026-04-10 03:15:19.661117" "level"=0 "msg"="creating resource group" "resourceGroup"="np-autoscaling-6x7zd5" STEP: creating cluster parameters @ 04/10/26 03:15:23.102 STEP: creating customer resources @ 04/10/26 03:15:23.102 STEP: creating the HCP cluster @ 04/10/26 03:16:46.624 "ts"="2026-04-10 03:16:46.624184" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-autoscale-cluster" "resourceGroup"="np-autoscaling-6x7zd5" STEP: verifying the cluster has default autoscaling parameters @ 04/10/26 03:27:40.525 STEP: getting admin credentials for the cluster @ 04/10/26 03:27:40.857 STEP: creating the AZ nodepool with 500 max replicas @ 04/10/26 03:36:33.03 STEP: verifying the AZ nodepool has the correct autoscaling configuration @ 04/10/26 03:46:22.114 STEP: verifying node count after AZ nodepool creation @ 04/10/26 03:46:30.42 STEP: updating the AZ nodepool max replicas from 500 to 4 before creating the next nodepool @ 04/10/26 03:46:31.463 STEP: verifying the AZ nodepool max replicas was updated to 4 @ 04/10/26 03:52:03.612 STEP: creating the no-AZ nodepool with 200 max replicas @ 04/10/26 03:52:03.976 STEP: verifying the no-AZ nodepool has the correct autoscaling configuration @ 04/10/26 04:01:52.182 STEP: verifying node count after no-AZ nodepool creation @ 04/10/26 04:01:53.049 "ts"="2026-04-10 04:01:54.362264" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:01:54.362332" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-pgwxc7" "resourceGroup"="np-autoscaling-6x7zd5" "ts"="2026-04-10 04:01:54.650082" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-pgwxc7" "resourceGroup"="" "ts"="2026-04-10 04:01:56.684085" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4c275018436d67b080599c8133799f947e5f8f174147f1d7309198de43b9be47.yaml.gz" "ts"="2026-04-10 04:01:56.684188" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4c275018436d67b080599c8133799f947e5f8f174147f1d7309198de43b9be47.yaml.gz" "ts"="2026-04-10 04:01:56.684258" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:01:56.685776" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-114" "ts"="2026-04-10 04:01:56.685793" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-6x7zd5" "ts"="2026-04-10 04:01:56.965161" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-6x7zd5" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs" "ts"="2026-04-10 04:01:57.199959" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-114" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:01:57.230486" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-6x7zd5" "deployment"="customer-infra-np-autoscale-cluster-pgwxc7" "ts"="2026-04-10 04:01:57.585359" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-114" "deployment"="msi-deployment-114" "ts"="2026-04-10 04:01:57.751304" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:01:57.751435" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:01:57.751569" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-6x7zd5" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 02:06:29 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc001134810>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-78qtcm, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-78qtcm" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-78qtcm, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc001134810>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-78qtcm, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-78qtcm" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-78qtcm, error: context deadline exceeded ... occurred Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775787080 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group [PreLaunchSetup:None, ARO-HCP-RP-API-Compatible, Importance:Critical, Positivity:Positive] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35 "ts"="2026-04-10 02:11:20.147352" "level"=0 "msg"="Starting identity container acquisition" "count"=3 "specID"="Customer-should-be-able-to-create-several-HCP-clusters-in-their-customer-resource-group,-but-not-in-the-same-managed-resource-group\|pid:123" "ts"="2026-04-10 02:11:20.304024" "level"=0 "msg"="Successfully assigned identity containers" "count"=3 "attempt"=1 "elapsed"="0s" STEP: creating a shared customer resource group @ 04/10/26 02:11:20.304 "ts"="2026-04-10 02:11:20.304178" "level"=0 "msg"="creating resource group" "resourceGroup"="customer-rg-rpmwqz" STEP: creating customer infrastructure and managed identities for first cluster @ 04/10/26 02:11:25.616 STEP: creating customer infrastructure and managed identities for second cluster @ 04/10/26 02:12:46.416 STEP: starting creation of both clusters in parallel @ 04/10/26 02:14:09.067 "ts"="2026-04-10 02:14:09.067714" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster" "resourceGroup"="customer-rg-rpmwqz" "ts"="2026-04-10 02:14:19.368932" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster2" "resourceGroup"="customer-rg-rpmwqz" STEP: waiting for first cluster to complete creation @ 04/10/26 02:14:28.135 STEP: waiting for second cluster to complete creation @ 04/10/26 02:45:48.132 Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775787080 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group [PreLaunchSetup:None, ARO-HCP-RP-API-Compatible, Importance:Critical, Positivity:Positive] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35 "ts"="2026-04-10 02:11:20.147352" "level"=0 "msg"="Starting identity container acquisition" "count"=3 "specID"="Customer-should-be-able-to-create-several-HCP-clusters-in-their-customer-resource-group,-but-not-in-the-same-managed-resource-group\|pid:123" "ts"="2026-04-10 02:11:20.304024" "level"=0 "msg"="Successfully assigned identity containers" "count"=3 "attempt"=1 "elapsed"="0s" STEP: creating a shared customer resource group @ 04/10/26 02:11:20.304 "ts"="2026-04-10 02:11:20.304178" "level"=0 "msg"="creating resource group" "resourceGroup"="customer-rg-rpmwqz" STEP: creating customer infrastructure and managed identities for first cluster @ 04/10/26 02:11:25.616 STEP: creating customer infrastructure and managed identities for second cluster @ 04/10/26 02:12:46.416 STEP: starting creation of both clusters in parallel @ 04/10/26 02:14:09.067 "ts"="2026-04-10 02:14:09.067714" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster" "resourceGroup"="customer-rg-rpmwqz" "ts"="2026-04-10 02:14:19.368932" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-hcp-cluster2" "resourceGroup"="customer-rg-rpmwqz" STEP: waiting for first cluster to complete creation @ 04/10/26 02:14:28.135 STEP: waiting for second cluster to complete creation @ 04/10/26 02:45:48.132 Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775792709 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41 "ts"="2026-04-10 03:45:09.057993" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-not-be-able-to-perform-various-invalid-operations-on-cluster-resources\|pid:930" "ts"="2026-04-10 03:45:09.065440" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 03:45:09.065 "ts"="2026-04-10 03:45:09.065489" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-szl4vm" STEP: creating cluster parameters @ 04/10/26 03:45:13.668 STEP: creating customer resources @ 04/10/26 03:45:13.668 STEP: creating the HCP cluster @ 04/10/26 03:46:32.455 "ts"="2026-04-10 03:46:32.456038" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-szl4vm" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775792709 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41 "ts"="2026-04-10 03:45:09.057993" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-not-be-able-to-perform-various-invalid-operations-on-cluster-resources\|pid:930" "ts"="2026-04-10 03:45:09.065440" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 03:45:09.065 "ts"="2026-04-10 03:45:09.065489" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-szl4vm" STEP: creating cluster parameters @ 04/10/26 03:45:13.668 STEP: creating customer resources @ 04/10/26 03:45:13.668 STEP: creating the HCP cluster @ 04/10/26 03:46:32.455 "ts"="2026-04-10 03:46:32.456038" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-szl4vm" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775791434 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 03:23:54.336107" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:917" "ts"="2026-04-10 03:23:54.342933" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 03:23:54.342 "ts"="2026-04-10 03:23:54.343003" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-6gpfxz" STEP: creating cluster parameters @ 04/10/26 03:23:59.225 STEP: creating customer resources @ 04/10/26 03:23:59.226 STEP: creating the HCP cluster @ 04/10/26 03:25:13.715 "ts"="2026-04-10 03:25:13.715863" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-6gpfxz" STEP: creating a storage account for shoebox logs @ 04/10/26 03:36:39.988 "ts"="2026-04-10 03:37:03.480045" "level"=0 "msg"="storage account created" "name"="shoebox5f8x7q" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-6gpfxz/providers/Microsoft.Storage/storageAccounts/shoebox5f8x7q" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 03:37:03.48 "ts"="2026-04-10 03:37:18.683593" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 03:37:18.683 "ts"="2026-04-10 04:04:19.006457" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoebox5f8x7q" "timestamp"="2026-04-10T04:04:19Z" "ts"="2026-04-10 04:04:19.006655" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:04:19.006753" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-72tcfz" "resourceGroup"="shoebox-cluster-6gpfxz" "ts"="2026-04-10 04:04:19.290758" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-72tcfz" "resourceGroup"="" "ts"="2026-04-10 04:04:21.555661" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 04:04:21.555765" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 04:04:21.555825" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:04:21.557374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-17" "ts"="2026-04-10 04:04:21.557391" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-6gpfxz" "ts"="2026-04-10 04:04:21.712755" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-17" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:04:21.844317" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6gpfxz" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 04:04:22.121298" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6gpfxz" "deployment"="customer-infra-shoebox-hcp-cluster-72tcfz" "ts"="2026-04-10 04:04:22.264844" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-17" "deployment"="msi-deployment-17" "ts"="2026-04-10 04:04:22.422889" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:04:22.423005" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:04:22.423122" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-6gpfxz" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775791434 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38 "ts"="2026-04-10 03:23:54.336107" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings\|pid:917" "ts"="2026-04-10 03:23:54.342933" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s" STEP: creating a resource group @ 04/10/26 03:23:54.342 "ts"="2026-04-10 03:23:54.343003" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-6gpfxz" STEP: creating cluster parameters @ 04/10/26 03:23:59.225 STEP: creating customer resources @ 04/10/26 03:23:59.226 STEP: creating the HCP cluster @ 04/10/26 03:25:13.715 "ts"="2026-04-10 03:25:13.715863" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-6gpfxz" STEP: creating a storage account for shoebox logs @ 04/10/26 03:36:39.988 "ts"="2026-04-10 03:37:03.480045" "level"=0 "msg"="storage account created" "name"="shoebox5f8x7q" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-6gpfxz/providers/Microsoft.Storage/storageAccounts/shoebox5f8x7q" STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 03:37:03.48 "ts"="2026-04-10 03:37:18.683593" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting" STEP: waiting for log containers to appear in the storage account @ 04/10/26 03:37:18.683 "ts"="2026-04-10 04:04:19.006457" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoebox5f8x7q" "timestamp"="2026-04-10T04:04:19Z" "ts"="2026-04-10 04:04:19.006655" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-10 04:04:19.006753" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-72tcfz" "resourceGroup"="shoebox-cluster-6gpfxz" "ts"="2026-04-10 04:04:19.290758" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-72tcfz" "resourceGroup"="" "ts"="2026-04-10 04:04:21.555661" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 04:04:21.555765" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz" "ts"="2026-04-10 04:04:21.555825" "level"=0 "msg"="collecting debug info" "ts"="2026-04-10 04:04:21.557374" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-17" "ts"="2026-04-10 04:04:21.557391" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-6gpfxz" "ts"="2026-04-10 04:04:21.712755" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-17" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-10 04:04:21.844317" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6gpfxz" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq" "ts"="2026-04-10 04:04:22.121298" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-6gpfxz" "deployment"="customer-infra-shoebox-hcp-cluster-72tcfz" "ts"="2026-04-10 04:04:22.264844" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-17" "deployment"="msi-deployment-17" "ts"="2026-04-10 04:04:22.422889" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-10 04:04:22.423005" "level"=0 "msg"="deleting created resources" "ts"="2026-04-10 04:04:22.423122" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-6gpfxz" Command Error: signal: killed Deserializaion Error: no output from command Deserializaion Error: no output from command 02:06:29 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure ontainers: requested 2 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 03:53:20.586689" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 03:54:20.589807" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 03:55:20.592550" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 03:56:20.596065" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 03:57:20.598781" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 03:58:20.601284" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 03:59:20.604055" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 04:00:20.606820" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 04:01:20.609895" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 04:02:20.612685" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 04:03:20.615863" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 04:04:20.618713" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 04:05:20.622636" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 04:06:20.624843" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 04:07:20.628107" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 04:08:20.630458" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=118 "retryIn"="1m0s" "elapsed"="1h57m0s" "ts"="2026-04-10 04:09:20.639551" "level"=0 "msg"="Successfully assigned identity containers" "count"=2 "attempt"=119 "elapsed"="1h58m0s" STEP: creating a resource group @ 04/10/26 04:09:20.639 "ts"="2026-04-10 04:09:20.639659" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-cluster-nsg-subnet-reuse-l4gfm9" STEP: creating customer resources @ 04/10/26 04:09:24.764 STEP: seeding HCP cluster @ 04/10/26 04:10:34.935 "ts"="2026-04-10 04:10:34.935456" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-cluster" "resourceGroup"="rg-cluster-nsg-subnet-reuse-l4gfm9" Command Error: signal: killed Deserializaion Error: no output from command ontainers: requested 2 identity containers but only 0 are assigned" "attempt"=102 "retryIn"="1m0s" "elapsed"="1h41m0s" "ts"="2026-04-10 03:53:20.586689" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=103 "retryIn"="1m0s" "elapsed"="1h42m0s" "ts"="2026-04-10 03:54:20.589807" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=104 "retryIn"="1m0s" "elapsed"="1h43m0s" "ts"="2026-04-10 03:55:20.592550" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=105 "retryIn"="1m0s" "elapsed"="1h44m0s" "ts"="2026-04-10 03:56:20.596065" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=106 "retryIn"="1m0s" "elapsed"="1h45m0s" "ts"="2026-04-10 03:57:20.598781" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=107 "retryIn"="1m0s" "elapsed"="1h46m0s" "ts"="2026-04-10 03:58:20.601284" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=108 "retryIn"="1m0s" "elapsed"="1h47m0s" "ts"="2026-04-10 03:59:20.604055" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 0 are assigned" "attempt"=109 "retryIn"="1m0s" "elapsed"="1h48m0s" "ts"="2026-04-10 04:00:20.606820" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=110 "retryIn"="1m0s" "elapsed"="1h49m0s" "ts"="2026-04-10 04:01:20.609895" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=111 "retryIn"="1m0s" "elapsed"="1h50m0s" "ts"="2026-04-10 04:02:20.612685" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=112 "retryIn"="1m0s" "elapsed"="1h51m0s" "ts"="2026-04-10 04:03:20.615863" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=113 "retryIn"="1m0s" "elapsed"="1h52m0s" "ts"="2026-04-10 04:04:20.618713" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=114 "retryIn"="1m0s" "elapsed"="1h53m0s" "ts"="2026-04-10 04:05:20.622636" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=115 "retryIn"="1m0s" "elapsed"="1h54m0s" "ts"="2026-04-10 04:06:20.624843" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=116 "retryIn"="1m0s" "elapsed"="1h55m0s" "ts"="2026-04-10 04:07:20.628107" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=117 "retryIn"="1m0s" "elapsed"="1h56m0s" "ts"="2026-04-10 04:08:20.630458" "level"=0 "msg"="Not enough free identity containers, waiting to retry" "error"="not enough free identity containers: requested 2 identity containers but only 1 are assigned" "attempt"=118 "retryIn"="1m0s" "elapsed"="1h57m0s" "ts"="2026-04-10 04:09:20.639551" "level"=0 "msg"="Successfully assigned identity containers" "count"=2 "attempt"=119 "elapsed"="1h58m0s" STEP: creating a resource group @ 04/10/26 04:09:20.639 "ts"="2026-04-10 04:09:20.639659" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-cluster-nsg-subnet-reuse-l4gfm9" STEP: creating customer resources @ 04/10/26 04:09:24.764 STEP: seeding HCP cluster @ 04/10/26 04:10:34.935 "ts"="2026-04-10 04:10:34.935456" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="basic-cluster" "resourceGroup"="rg-cluster-nsg-subnet-reuse-l4gfm9" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 02:06:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc00096a480>: failed waiting for hcpCluster="cluster-stable-4-20-hcbxqh" in resourcegroup="rg-stable-4-20-tpn8xl" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-hcbxqh in resource group rg-stable-4-20-tpn8xl, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:129]: Unexpected error: <fmt.wrapErrors \| 0xc00096a480>: failed waiting for hcpCluster="cluster-stable-4-20-hcbxqh" in resourcegroup="rg-stable-4-20-tpn8xl" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-hcbxqh in resource group rg-stable-4-20-tpn8xl, error: context deadline exceeded ... occurred unknown (1) Process did not finish before 2h0m0s timeout 02:06:29 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T04:11:17Z"}
02:00:34	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a