CIHealth Run Log

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
18:45:33	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	14	e2e (13) Deserializaion Error: no output from command 18:45:33 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775245707 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237 STEP: creating resource group @ 04/03/26 19:48:28.242 "ts"="2026-04-03 19:48:28.242687" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" STEP: creating cluster parameters at control plane version @ 04/03/26 19:48:32.172 STEP: creating customer resources @ 04/03/26 19:48:32.273 STEP: creating the HCP cluster with version 4.20.17 @ 04/03/26 19:49:34.027 "ts"="2026-04-03 19:49:34.027707" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gh68jv" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 20:01:59.339 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 20:11:15.755 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 20:13:15.571 STEP: capturing node release images before upgrade @ 04/03/26 20:13:15.748 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/03/26 20:13:15.951 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 20:14:02.797 STEP: verifying node pool GET still reflects the new version @ 04/03/26 20:42:11.233 "ts"="2026-04-03 20:42:12.161810" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 20:42:12.161939" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-gh68jv-rmkr2d" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "ts"="2026-04-03 20:42:12.456525" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-gh68jv-rmkr2d" "resourceGroup"="" "ts"="2026-04-03 20:42:14.379525" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-03 20:42:14.379643" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-03 20:42:14.379720" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 20:42:14.381290" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-45" "ts"="2026-04-03 20:42:14.381308" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "ts"="2026-04-03 20:42:14.597243" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "deployment"="pooledNonMsiScopedAssignments-gyhigscniinga" "ts"="2026-04-03 20:42:14.870794" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-45" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 20:42:14.912330" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "deployment"="customer-infra-np-version-upgrade-cluster-gh68jv-rmkr2d" "ts"="2026-04-03 20:42:15.466293" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-45" "deployment"="msi-deployment-45" "ts"="2026-04-03 20:42:15.973131" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 20:42:15.973254" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 20:42:15.973386" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775245707 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237 STEP: creating resource group @ 04/03/26 19:48:28.242 "ts"="2026-04-03 19:48:28.242687" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" STEP: creating cluster parameters at control plane version @ 04/03/26 19:48:32.172 STEP: creating customer resources @ 04/03/26 19:48:32.273 STEP: creating the HCP cluster with version 4.20.17 @ 04/03/26 19:49:34.027 "ts"="2026-04-03 19:49:34.027707" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gh68jv" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 20:01:59.339 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 20:11:15.755 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 20:13:15.571 STEP: capturing node release images before upgrade @ 04/03/26 20:13:15.748 STEP: triggering nodepool upgrade to version 4.20.17 @ 04/03/26 20:13:15.951 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 20:14:02.797 STEP: verifying node pool GET still reflects the new version @ 04/03/26 20:42:11.233 "ts"="2026-04-03 20:42:12.161810" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 20:42:12.161939" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-gh68jv-rmkr2d" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "ts"="2026-04-03 20:42:12.456525" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-gh68jv-rmkr2d" "resourceGroup"="" "ts"="2026-04-03 20:42:14.379525" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-03 20:42:14.379643" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz" "ts"="2026-04-03 20:42:14.379720" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 20:42:14.381290" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-45" "ts"="2026-04-03 20:42:14.381308" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "ts"="2026-04-03 20:42:14.597243" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "deployment"="pooledNonMsiScopedAssignments-gyhigscniinga" "ts"="2026-04-03 20:42:14.870794" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-45" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 20:42:14.912330" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" "deployment"="customer-infra-np-version-upgrade-cluster-gh68jv-rmkr2d" "ts"="2026-04-03 20:42:15.466293" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-45" "deployment"="msi-deployment-45" "ts"="2026-04-03 20:42:15.973131" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 20:42:15.973254" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 20:42:15.973386" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-gh68jv-8kf8pf" Command Error: signal: killed Deserializaion Error: no output from command Interrupted by User 18:45:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User Deserializaion Error: no output from command 18:45:33 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure "="0" "beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="b0t7x8b0g1n2w3n-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "kubernetes.io/os"="linux" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="switzerlandnorth" "kubernetes.io/hostname"="b0t7x8b0g1n2w3n-np-1-wqrsn-fsz9k" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/region"="switzerlandnorth"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 20:34:41.587768" "level"=0 "msg"="Node" "name"="b0t7x8b0g1n2w3n-np-1-wqrsn-hzcv6" "labels"={"beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="switzerlandnorth" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/os"="linux" "node.openshift.io/os_id"="rhel" "hypershift.openshift.io/nodePool"="b0t7x8b0g1n2w3n-np-1" "topology.kubernetes.io/zone"="1" "failure-domain.beta.kubernetes.io/zone"="1" "key1"="value1" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="b0t7x8b0g1n2w3n-np-1-wqrsn-hzcv6" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/managed"="true" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="switzerlandnorth" "beta.kubernetes.io/arch"="amd64"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 20:34:41.587818" "level"=0 "msg"="Node" "name"="b0t7x8b0g1n2w3n-np-1-wqrsn-k6pvw" "labels"={"beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="switzerlandnorth" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="b0t7x8b0g1n2w3n-np-1-wqrsn-k6pvw" "node-role.kubernetes.io/worker"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="b0t7x8b0g1n2w3n-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/os"="linux" "key1"="value1" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="switzerlandnorth" "failure-domain.beta.kubernetes.io/zone"="0"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/03/26 20:34:41.587 STEP: verifying original taints persist on 2 nodes @ 04/03/26 20:34:41.587 "ts"="2026-04-03 20:34:41.588122" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 20:34:41.588264" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-44fz9p" "resourceGroup"="rg-np-labels-taints-4z7w6h" "ts"="2026-04-03 20:34:41.767633" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-44fz9p" "resourceGroup"="" "ts"="2026-04-03 20:34:43.091201" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 20:34:43.091299" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 20:34:43.091358" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 20:34:43.092891" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-96" "ts"="2026-04-03 20:34:43.092910" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-4z7w6h" "ts"="2026-04-03 20:34:43.282140" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-4z7w6h" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-03 20:34:43.445177" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-96" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 20:34:43.456328" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-4z7w6h" "deployment"="customer-infra-cluster-np-labels-taints-44fz9p" "ts"="2026-04-03 20:34:43.747143" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-96" "deployment"="msi-deployment-96" "ts"="2026-04-03 20:34:44.098708" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 20:34:44.098863" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 20:34:44.098996" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-4z7w6h" Command Error: signal: killed Deserializaion Error: no output from command "="0" "beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="b0t7x8b0g1n2w3n-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "kubernetes.io/os"="linux" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/region"="switzerlandnorth" "kubernetes.io/hostname"="b0t7x8b0g1n2w3n-np-1-wqrsn-fsz9k" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/region"="switzerlandnorth"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 20:34:41.587768" "level"=0 "msg"="Node" "name"="b0t7x8b0g1n2w3n-np-1-wqrsn-hzcv6" "labels"={"beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="switzerlandnorth" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/os"="linux" "node.openshift.io/os_id"="rhel" "hypershift.openshift.io/nodePool"="b0t7x8b0g1n2w3n-np-1" "topology.kubernetes.io/zone"="1" "failure-domain.beta.kubernetes.io/zone"="1" "key1"="value1" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="b0t7x8b0g1n2w3n-np-1-wqrsn-hzcv6" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/managed"="true" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="switzerlandnorth" "beta.kubernetes.io/arch"="amd64"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 20:34:41.587818" "level"=0 "msg"="Node" "name"="b0t7x8b0g1n2w3n-np-1-wqrsn-k6pvw" "labels"={"beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="switzerlandnorth" "kubernetes.io/arch"="amd64" "kubernetes.io/hostname"="b0t7x8b0g1n2w3n-np-1-wqrsn-k6pvw" "node-role.kubernetes.io/worker"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="b0t7x8b0g1n2w3n-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/os"="linux" "key1"="value1" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="switzerlandnorth" "failure-domain.beta.kubernetes.io/zone"="0"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/03/26 20:34:41.587 STEP: verifying original taints persist on 2 nodes @ 04/03/26 20:34:41.587 "ts"="2026-04-03 20:34:41.588122" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 20:34:41.588264" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-44fz9p" "resourceGroup"="rg-np-labels-taints-4z7w6h" "ts"="2026-04-03 20:34:41.767633" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-44fz9p" "resourceGroup"="" "ts"="2026-04-03 20:34:43.091201" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 20:34:43.091299" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 20:34:43.091358" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 20:34:43.092891" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-96" "ts"="2026-04-03 20:34:43.092910" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-4z7w6h" "ts"="2026-04-03 20:34:43.282140" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-4z7w6h" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-03 20:34:43.445177" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-96" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 20:34:43.456328" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-4z7w6h" "deployment"="customer-infra-cluster-np-labels-taints-44fz9p" "ts"="2026-04-03 20:34:43.747143" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-96" "deployment"="msi-deployment-96" "ts"="2026-04-03 20:34:44.098708" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 20:34:44.098863" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 20:34:44.098996" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-4z7w6h" Command Error: signal: killed Deserializaion Error: no output from command Interrupted by User 18:45:33 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User Interrupted by User 18:45:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 18:45:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0013f0840>: failed waiting for hcpCluster="cluster-zstream-4-19-f4hz5d" in resourcegroup="rg-zstream-upgrade-4-19-mf7hcn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-f4hz5d\" in resourcegroup=\"rg-zstream-upgrade-4-19-mf7hcn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015a34a0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0013f0840>: failed waiting for hcpCluster="cluster-zstream-4-19-f4hz5d" in resourcegroup="rg-zstream-upgrade-4-19-mf7hcn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-f4hz5d\" in resourcegroup=\"rg-zstream-upgrade-4-19-mf7hcn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015a34a0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-f4hz5d in resource group rg-zstream-upgrade-4-19-mf7hcn", }, <context.deadlineExceededError>{}, ], } occurred Interrupted by User 18:45:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User Deserializaion Error: no output from command 18:45:33 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775245500 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 STEP: creating resource group @ 04/03/26 19:45:00.935 "ts"="2026-04-03 19:45:00.935972" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" STEP: creating cluster parameters at control plane version @ 04/03/26 19:45:03.992 STEP: creating customer resources @ 04/03/26 19:45:04.081 STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 19:46:06.079 "ts"="2026-04-03 19:46:06.079283" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-pldnh4" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 19:59:38.878 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 20:09:14.928 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 20:11:13.489 STEP: capturing node release images before upgrade @ 04/03/26 20:11:13.637 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 20:11:13.841 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 20:12:07.744 STEP: verifying node pool GET still reflects the new version @ 04/03/26 20:40:16.114 "ts"="2026-04-03 20:40:17.269118" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 20:40:17.269207" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-pldnh4-8gpfcs" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "ts"="2026-04-03 20:40:17.673955" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-pldnh4-8gpfcs" "resourceGroup"="" "ts"="2026-04-03 20:40:19.163768" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 20:40:19.163906" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 20:40:19.163979" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 20:40:19.165684" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-47" "ts"="2026-04-03 20:40:19.165685" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "ts"="2026-04-03 20:40:19.357542" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "deployment"="pooledNonMsiScopedAssignments-7onhmt3j3ybeg" "ts"="2026-04-03 20:40:19.452634" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-47" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 20:40:19.512189" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "deployment"="customer-infra-np-version-upgrade-cluster-pldnh4-8gpfcs" "ts"="2026-04-03 20:40:19.735871" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-47" "deployment"="msi-deployment-47" "ts"="2026-04-03 20:40:19.871923" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 20:40:19.872029" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 20:40:19.872148" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775245500 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 STEP: creating resource group @ 04/03/26 19:45:00.935 "ts"="2026-04-03 19:45:00.935972" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" STEP: creating cluster parameters at control plane version @ 04/03/26 19:45:03.992 STEP: creating customer resources @ 04/03/26 19:45:04.081 STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 19:46:06.079 "ts"="2026-04-03 19:46:06.079283" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-pldnh4" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 19:59:38.878 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 20:09:14.928 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 20:11:13.489 STEP: capturing node release images before upgrade @ 04/03/26 20:11:13.637 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 20:11:13.841 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 20:12:07.744 STEP: verifying node pool GET still reflects the new version @ 04/03/26 20:40:16.114 "ts"="2026-04-03 20:40:17.269118" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 20:40:17.269207" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-pldnh4-8gpfcs" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "ts"="2026-04-03 20:40:17.673955" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-pldnh4-8gpfcs" "resourceGroup"="" "ts"="2026-04-03 20:40:19.163768" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 20:40:19.163906" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 20:40:19.163979" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 20:40:19.165684" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-47" "ts"="2026-04-03 20:40:19.165685" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "ts"="2026-04-03 20:40:19.357542" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "deployment"="pooledNonMsiScopedAssignments-7onhmt3j3ybeg" "ts"="2026-04-03 20:40:19.452634" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-47" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 20:40:19.512189" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" "deployment"="customer-infra-np-version-upgrade-cluster-pldnh4-8gpfcs" "ts"="2026-04-03 20:40:19.735871" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-47" "deployment"="msi-deployment-47" "ts"="2026-04-03 20:40:19.871923" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 20:40:19.872029" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 20:40:19.872148" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-pldnh4-hs5cqs" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 18:45:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc0011b0810>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2v4phv" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-2v4phv\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0009a5550>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc0011b0810>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2v4phv" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-2v4phv\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0009a5550>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-2v4phv", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 18:45:33 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapError \| 0xc0004e0160>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w8b9cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w8b9cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015f0180>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Correlation-Request-Id": [ "ea03dd36-95b8-47c7-90e1-a8dc79a919d8", ], "X-Content-Type-Options": ["nosniff"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "09d15fc6-6c8e-4784-845a-952eadfe9277", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T193832Z:ea03dd36-95b8-47c7-90e1-a8dc79a919d8", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: EFBD4EB481774F60AB6EA1EDCD5ECFAF Ref B: CHI261140508025 Ref C: 2026-04-03T19:38:32Z", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["80"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/6e183501-6da1-4585-8100-e8772f21fd65", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Date": [ "Fri, 03 Apr 2026 19:38:32 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0004e0000>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error: <fmt.wrapError \| 0xc0004e0160>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w8b9cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-w8b9cs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015f0180>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Correlation-Request-Id": [ "ea03dd36-95b8-47c7-90e1-a8dc79a919d8", ], "X-Content-Type-Options": ["nosniff"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "09d15fc6-6c8e-4784-845a-952eadfe9277", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T193832Z:ea03dd36-95b8-47c7-90e1-a8dc79a919d8", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: EFBD4EB481774F60AB6EA1EDCD5ECFAF Ref B: CHI261140508025 Ref C: 2026-04-03T19:38:32Z", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["80"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/6e183501-6da1-4585-8100-e8772f21fd65", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Date": [ "Fri, 03 Apr 2026 19:38:32 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0004e0000>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 18:45:33 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc000eb6a20>: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-l56s4b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-l56s4b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000905dd0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc000eb6a20>: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-l56s4b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-l56s4b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000905dd0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-l56s4b", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 18:45:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error: <fmt.wrapError \| 0xc000ecca20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-kfr69r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-kfr69r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012090e0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "c90c7b13-2d84-49c6-9e50-7c4041ceed5c", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "Date": [ "Fri, 03 Apr 2026 19:32:05 GMT", ], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["151"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T193206Z:209e4973-2412-46de-aeb0-3a5184e5b37d", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/1c2454d9-3f36-4903-bf4d-7d6a91e76c4f", ], "X-Msedge-Ref": [ "Ref A: DA3E32013CFF4BB48925BAB35C81373F Ref B: CHI261140504060 Ref C: 2026-04-03T19:32:05Z", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Correlation-Request-Id": [ "209e4973-2412-46de-aeb0-3a5184e5b37d", ], }, Body: <exported.nopClosingBytesReader \| 0xc000ecc960>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error: <fmt.wrapError \| 0xc000ecca20>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-kfr69r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-kfr69r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012090e0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "c90c7b13-2d84-49c6-9e50-7c4041ceed5c", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "Date": [ "Fri, 03 Apr 2026 19:32:05 GMT", ], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["151"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T193206Z:209e4973-2412-46de-aeb0-3a5184e5b37d", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/1c2454d9-3f36-4903-bf4d-7d6a91e76c4f", ], "X-Msedge-Ref": [ "Ref A: DA3E32013CFF4BB48925BAB35C81373F Ref B: CHI261140504060 Ref C: 2026-04-03T19:32:05Z", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Correlation-Request-Id": [ "209e4973-2412-46de-aeb0-3a5184e5b37d", ], }, Body: <exported.nopClosingBytesReader \| 0xc000ecc960>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 18:45:33 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapError \| 0xc000988ce0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-bx2wzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-cq5b22/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-bx2wzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-cq5b22/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000def680>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Msedge-Ref": [ "Ref A: DAC0D39B3291499EB510E196A8464291 Ref B: CHI261140504052 Ref C: 2026-04-03T19:47:34Z", ], "X-Ms-Request-Id": [ "bbeb88a5-59d3-41b8-9d97-5ea6acb0a3e2", ], "X-Envoy-Upstream-Service-Time": ["497"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 03 Apr 2026 19:47:34 GMT", ], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/704ea369-be04-459b-a6e0-8571e30af23b", ], "X-Content-Type-Options": ["nosniff"], "Pragma": ["no-cache"], "X-Ms-Correlation-Request-Id": [ "5d4a3adb-fb62-43e1-82b6-3016b7ae4e34", ], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T194735Z:5d4a3adb-fb62-43e1-82b6-3016b7ae4e34", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Expires": ["-1"], }, Body: <exported.nopClosingBytesReader \| 0xc000988c20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapError \| 0xc000988ce0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-bx2wzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-cq5b22/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-bx2wzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-cq5b22/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000def680>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Msedge-Ref": [ "Ref A: DAC0D39B3291499EB510E196A8464291 Ref B: CHI261140504052 Ref C: 2026-04-03T19:47:34Z", ], "X-Ms-Request-Id": [ "bbeb88a5-59d3-41b8-9d97-5ea6acb0a3e2", ], "X-Envoy-Upstream-Service-Time": ["497"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 03 Apr 2026 19:47:34 GMT", ], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/704ea369-be04-459b-a6e0-8571e30af23b", ], "X-Content-Type-Options": ["nosniff"], "Pragma": ["no-cache"], "X-Ms-Correlation-Request-Id": [ "5d4a3adb-fb62-43e1-82b6-3016b7ae4e34", ], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T194735Z:5d4a3adb-fb62-43e1-82b6-3016b7ae4e34", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Expires": ["-1"], }, Body: <exported.nopClosingBytesReader \| 0xc000988c20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred unknown (1) Process did not finish before 2h0m0s timeout 18:45:33 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-03T20:48:28Z"}
15:33:14	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	6	e2e (6) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 15:33:14 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapError \| 0xc001312be0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-lzprzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-lzprzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012997d0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Msedge-Ref": [ "Ref A: 816BEB550F724B7CB369966EADF99F38 Ref B: CHI261140512042 Ref C: 2026-04-03T16:43:20Z", ], "Pragma": ["no-cache"], "X-Ms-Request-Id": [ "7a3d0639-8fbf-4c85-a9ac-cf8fd21633c5", ], "X-Envoy-Upstream-Service-Time": ["126"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260403T164320Z:c9868f2c-f75d-44e0-a3df-da965aea61d0", ], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/36c5491e-caaa-456e-b0f7-93cb3a0d8078", ], "X-Ms-Correlation-Request-Id": [ "c9868f2c-f75d-44e0-a3df-da965aea61d0", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 03 Apr 2026 16:43:20 GMT", ], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Content-Type-Options": ["nosniff"], }, Body: <exported.nopClosingBytesReader \| 0xc001312a60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "ma... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error: <fmt.wrapError \| 0xc001312be0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-lzprzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-lzprzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012997d0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Msedge-Ref": [ "Ref A: 816BEB550F724B7CB369966EADF99F38 Ref B: CHI261140512042 Ref C: 2026-04-03T16:43:20Z", ], "Pragma": ["no-cache"], "X-Ms-Request-Id": [ "7a3d0639-8fbf-4c85-a9ac-cf8fd21633c5", ], "X-Envoy-Upstream-Service-Time": ["126"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260403T164320Z:c9868f2c-f75d-44e0-a3df-da965aea61d0", ], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/36c5491e-caaa-456e-b0f7-93cb3a0d8078", ], "X-Ms-Correlation-Request-Id": [ "c9868f2c-f75d-44e0-a3df-da965aea61d0", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 03 Apr 2026 16:43:20 GMT", ], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Content-Type-Options": ["nosniff"], }, Body: <exported.nopClosingBytesReader \| 0xc001312a60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "ma... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Interrupted by User 15:33:14 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:51]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:51]: Interrupted by User timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 15:33:14 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc000a56690>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-kgcl9v" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-kgcl9v, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-kgcl9v\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-kgcl9v, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001144180>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-kgcl9v", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc000a56690>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-kgcl9v" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-kgcl9v, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-kgcl9v\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-kgcl9v, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001144180>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-kgcl9v", }, <context.deadlineExceededError>{}, ], } occurred Deserializaion Error: no output from command 15:33:14 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775234199 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 STEP: creating resource group @ 04/03/26 16:36:42.47 "ts"="2026-04-03 16:36:42.470300" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gp64gv-cl96bf" STEP: creating cluster parameters at control plane version @ 04/03/26 16:36:46.204 STEP: creating customer resources @ 04/03/26 16:36:46.308 STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 16:37:39.379 "ts"="2026-04-03 16:37:39.379203" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gp64gv" "resourceGroup"="rg-np-version-upgrade-gp64gv-cl96bf" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 16:53:54.034 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 17:06:34.387 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 17:10:31.018 STEP: capturing node release images before upgrade @ 04/03/26 17:10:31.157 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 17:10:31.206 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 17:11:12.494 Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775234199 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 STEP: creating resource group @ 04/03/26 16:36:42.47 "ts"="2026-04-03 16:36:42.470300" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gp64gv-cl96bf" STEP: creating cluster parameters at control plane version @ 04/03/26 16:36:46.204 STEP: creating customer resources @ 04/03/26 16:36:46.308 STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 16:37:39.379 "ts"="2026-04-03 16:37:39.379203" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gp64gv" "resourceGroup"="rg-np-version-upgrade-gp64gv-cl96bf" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 16:53:54.034 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 17:06:34.387 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 17:10:31.018 STEP: capturing node release images before upgrade @ 04/03/26 17:10:31.157 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 17:10:31.206 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 17:11:12.494 Command Error: signal: killed Deserializaion Error: no output from command ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 15:33:14 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error: <fmt.wrapError \| 0xc000ed3be0>: failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-k2sdw2" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-k2sdw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-k2sdw2\" for cluster=\"ea-cluster\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-k2sdw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000990240>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: FCB0395A816E4F088A6267F3F9575104 Ref B: CH1AA2020603037 Ref C: 2026-04-03T16:45:56Z", ], "Pragma": ["no-cache"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Correlation-Request-Id": [ "5665c40b-cd9d-496e-b953-e6902aa7df2a", ], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["207"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/e67cee86-8970-401d-ad15-7fe738de3eb4", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260403T164556Z:5665c40b-cd9d-496e-b953-e6902aa7df2a", ], "Date": [ "Fri, 03 Apr 2026 16:45:56 GMT", ], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "1bcce103-7d8c-43c7-af0c-2c2897018d90", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], }, Body: <exported.nopClosingBytesReader \| 0xc000ed3b20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:172]: Unexpected error: <fmt.wrapError \| 0xc000ed3be0>: failed creating external auth "external-auth" in resourcegroup="external-auth-cluster-k2sdw2" for cluster="ea-cluster": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-k2sdw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed creating external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-k2sdw2\" for cluster=\"ea-cluster\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-k2sdw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster/externalAuths/external-auth\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000990240>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: FCB0395A816E4F088A6267F3F9575104 Ref B: CH1AA2020603037 Ref C: 2026-04-03T16:45:56Z", ], "Pragma": ["no-cache"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Correlation-Request-Id": [ "5665c40b-cd9d-496e-b953-e6902aa7df2a", ], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["207"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/e67cee86-8970-401d-ad15-7fe738de3eb4", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260403T164556Z:5665c40b-cd9d-496e-b953-e6902aa7df2a", ], "Date": [ "Fri, 03 Apr 2026 16:45:56 GMT", ], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "1bcce103-7d8c-43c7-af0c-2c2897018d90", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], }, Body: <exported.nopClosingBytesReader \| 0xc000ed3b20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Deserializaion Error: no output from command 15:33:14 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure ain.beta.kubernetes.io/region"="canadacentral" "kubernetes.io/arch"="amd64" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="1" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="b9c8m2q2v1t3y4d-np-1" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="canadacentral" "topology.kubernetes.io/zone"="1" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "kubernetes.io/hostname"="b9c8m2q2v1t3y4d-np-1-vfvt7-pbgf2" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "kubernetes.io/os"="linux" "node.openshift.io/os_id"="rhel"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 17:30:39.295240" "level"=0 "msg"="Node" "name"="b9c8m2q2v1t3y4d-np-1-vfvt7-rhnzh" "labels"={"kubernetes.io/hostname"="b9c8m2q2v1t3y4d-np-1-vfvt7-rhnzh" "topology.kubernetes.io/zone"="0" "failure-domain.beta.kubernetes.io/region"="canadacentral" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/zone"="0" "kubernetes.io/arch"="amd64" "kubernetes.io/os"="linux" "hypershift.openshift.io/nodePool"="b9c8m2q2v1t3y4d-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "node-role.kubernetes.io/worker"="" "topology.kubernetes.io/region"="canadacentral" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/managed"="true"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 17:30:39.295287" "level"=0 "msg"="Node" "name"="b9c8m2q2v1t3y4d-np-1-vfvt7-rqpwz" "labels"={"failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.disk.csi.azure.com/zone"="" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "kubernetes.io/os"="linux" "node-role.kubernetes.io/worker"="" "failure-domain.beta.kubernetes.io/region"="canadacentral" "key2"="value2" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/region"="canadacentral" "hypershift.openshift.io/nodePool"="b9c8m2q2v1t3y4d-np-1" "kubernetes.io/hostname"="b9c8m2q2v1t3y4d-np-1-vfvt7-rqpwz" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/03/26 17:30:39.295 STEP: verifying original taints persist on 2 nodes @ 04/03/26 17:30:39.295 "ts"="2026-04-03 17:30:39.295520" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 17:30:39.295601" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-7gmvg5" "resourceGroup"="rg-np-labels-taints-6cz9nd" "ts"="2026-04-03 17:30:39.369808" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-7gmvg5" "resourceGroup"="" "ts"="2026-04-03 17:30:39.832363" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 17:30:39.832500" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 17:30:39.832588" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 17:30:39.835263" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "ts"="2026-04-03 17:30:39.835280" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-6cz9nd" "ts"="2026-04-03 17:30:39.919508" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-6cz9nd" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-03 17:30:39.981371" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-6cz9nd" "deployment"="customer-infra-cluster-np-labels-taints-7gmvg5" "ts"="2026-04-03 17:30:40.314152" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 17:30:40.654016" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="msi-deployment-4" "ts"="2026-04-03 17:30:40.790499" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 17:30:40.790625" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 17:30:40.790729" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-6cz9nd" Command Error: signal: killed Deserializaion Error: no output from command ain.beta.kubernetes.io/region"="canadacentral" "kubernetes.io/arch"="amd64" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/zone"="1" "hypershift.openshift.io/managed"="true" "hypershift.openshift.io/nodePool"="b9c8m2q2v1t3y4d-np-1" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="canadacentral" "topology.kubernetes.io/zone"="1" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "kubernetes.io/hostname"="b9c8m2q2v1t3y4d-np-1-vfvt7-pbgf2" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "kubernetes.io/os"="linux" "node.openshift.io/os_id"="rhel"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 17:30:39.295240" "level"=0 "msg"="Node" "name"="b9c8m2q2v1t3y4d-np-1-vfvt7-rhnzh" "labels"={"kubernetes.io/hostname"="b9c8m2q2v1t3y4d-np-1-vfvt7-rhnzh" "topology.kubernetes.io/zone"="0" "failure-domain.beta.kubernetes.io/region"="canadacentral" "node.openshift.io/os_id"="rhel" "topology.disk.csi.azure.com/zone"="" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "failure-domain.beta.kubernetes.io/zone"="0" "kubernetes.io/arch"="amd64" "kubernetes.io/os"="linux" "hypershift.openshift.io/nodePool"="b9c8m2q2v1t3y4d-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "key1"="value1" "node-role.kubernetes.io/worker"="" "topology.kubernetes.io/region"="canadacentral" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/managed"="true"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}] "ts"="2026-04-03 17:30:39.295287" "level"=0 "msg"="Node" "name"="b9c8m2q2v1t3y4d-np-1-vfvt7-rqpwz" "labels"={"failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.disk.csi.azure.com/zone"="" "beta.kubernetes.io/arch"="amd64" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "kubernetes.io/os"="linux" "node-role.kubernetes.io/worker"="" "failure-domain.beta.kubernetes.io/region"="canadacentral" "key2"="value2" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/region"="canadacentral" "hypershift.openshift.io/nodePool"="b9c8m2q2v1t3y4d-np-1" "kubernetes.io/hostname"="b9c8m2q2v1t3y4d-np-1-vfvt7-rqpwz" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}] STEP: verifying original labels persist on 3 nodes @ 04/03/26 17:30:39.295 STEP: verifying original taints persist on 2 nodes @ 04/03/26 17:30:39.295 "ts"="2026-04-03 17:30:39.295520" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 17:30:39.295601" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-7gmvg5" "resourceGroup"="rg-np-labels-taints-6cz9nd" "ts"="2026-04-03 17:30:39.369808" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-7gmvg5" "resourceGroup"="" "ts"="2026-04-03 17:30:39.832363" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 17:30:39.832500" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz" "ts"="2026-04-03 17:30:39.832588" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 17:30:39.835263" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "ts"="2026-04-03 17:30:39.835280" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-6cz9nd" "ts"="2026-04-03 17:30:39.919508" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-6cz9nd" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k" "ts"="2026-04-03 17:30:39.981371" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-6cz9nd" "deployment"="customer-infra-cluster-np-labels-taints-7gmvg5" "ts"="2026-04-03 17:30:40.314152" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 17:30:40.654016" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-4" "deployment"="msi-deployment-4" "ts"="2026-04-03 17:30:40.790499" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 17:30:40.790625" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 17:30:40.790729" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-6cz9nd" Command Error: signal: killed Deserializaion Error: no output from command
06:22:01	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	9	e2e (9) timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:22:01 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc000d28cc0>: failed waiting for hcpCluster="cluster-stable-4-20-cg7svw" in resourcegroup="rg-stable-4-20-vkp9t8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-cg7svw in resource group rg-stable-4-20-vkp9t8, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-cg7svw\" in resourcegroup=\"rg-stable-4-20-vkp9t8\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-cg7svw in resource group rg-stable-4-20-vkp9t8, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00081b460>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-cg7svw in resource group rg-stable-4-20-vkp9t8", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc000d28cc0>: failed waiting for hcpCluster="cluster-stable-4-20-cg7svw" in resourcegroup="rg-stable-4-20-vkp9t8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-cg7svw in resource group rg-stable-4-20-vkp9t8, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-cg7svw\" in resourcegroup=\"rg-stable-4-20-vkp9t8\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-cg7svw in resource group rg-stable-4-20-vkp9t8, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00081b460>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-cg7svw in resource group rg-stable-4-20-vkp9t8", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:22:01 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc001816030>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-s8b4sw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-s8b4sw, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s8b4sw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-s8b4sw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000763cd0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-s8b4sw", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc001816030>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-s8b4sw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-s8b4sw, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s8b4sw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-s8b4sw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000763cd0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-s8b4sw", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:22:01 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc00078a450>: failed waiting for hcpCluster="admin-cred-lifecycle-5wntpm" in resourcegroup="admin-credential-lifecycle-test-5h6742" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wntpm in resource group admin-credential-lifecycle-test-5h6742, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-5wntpm\" in resourcegroup=\"admin-credential-lifecycle-test-5h6742\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wntpm in resource group admin-credential-lifecycle-test-5h6742, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f442c0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wntpm in resource group admin-credential-lifecycle-test-5h6742", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc00078a450>: failed waiting for hcpCluster="admin-cred-lifecycle-5wntpm" in resourcegroup="admin-credential-lifecycle-test-5h6742" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wntpm in resource group admin-credential-lifecycle-test-5h6742, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-5wntpm\" in resourcegroup=\"admin-credential-lifecycle-test-5h6742\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wntpm in resource group admin-credential-lifecycle-test-5h6742, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f442c0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-5wntpm in resource group admin-credential-lifecycle-test-5h6742", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:22:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0006cb290>: failed waiting for hcpCluster="cluster-zstream-4-19-xw78lr" in resourcegroup="rg-zstream-upgrade-4-19-xdqmmk" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-xw78lr in resource group rg-zstream-upgrade-4-19-xdqmmk, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-xw78lr\" in resourcegroup=\"rg-zstream-upgrade-4-19-xdqmmk\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-xw78lr in resource group rg-zstream-upgrade-4-19-xdqmmk, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e85f30>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-xw78lr in resource group rg-zstream-upgrade-4-19-xdqmmk", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0006cb290>: failed waiting for hcpCluster="cluster-zstream-4-19-xw78lr" in resourcegroup="rg-zstream-upgrade-4-19-xdqmmk" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-xw78lr in resource group rg-zstream-upgrade-4-19-xdqmmk, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-xw78lr\" in resourcegroup=\"rg-zstream-upgrade-4-19-xdqmmk\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-xw78lr in resource group rg-zstream-upgrade-4-19-xdqmmk, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e85f30>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-xw78lr in resource group rg-zstream-upgrade-4-19-xdqmmk", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:22:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc000bbc060>: failed waiting for hcpCluster="cluster-zstream-4-20-92cq46" in resourcegroup="rg-zstream-upgrade-4-20-76fppt" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-92cq46 in resource group rg-zstream-upgrade-4-20-76fppt, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-92cq46\" in resourcegroup=\"rg-zstream-upgrade-4-20-76fppt\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-92cq46 in resource group rg-zstream-upgrade-4-20-76fppt, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00064ef70>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-92cq46 in resource group rg-zstream-upgrade-4-20-76fppt", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc000bbc060>: failed waiting for hcpCluster="cluster-zstream-4-20-92cq46" in resourcegroup="rg-zstream-upgrade-4-20-76fppt" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-92cq46 in resource group rg-zstream-upgrade-4-20-76fppt, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-92cq46\" in resourcegroup=\"rg-zstream-upgrade-4-20-76fppt\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-92cq46 in resource group rg-zstream-upgrade-4-20-76fppt, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00064ef70>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-92cq46 in resource group rg-zstream-upgrade-4-20-76fppt", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 06:22:01 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error: <fmt.wrapError \| 0xc000a92c60>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ml5zmp-22l25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ml5zmp/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ml5zmp-22l25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ml5zmp/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000b47860>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/a147bbb3-95ba-46a8-99a0-b2bba5e615c9", ], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "Expires": ["-1"], "Cache-Control": ["no-cache"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T074559Z:00b9e938-05fe-4cc6-bc31-bd2cf021e23c", ], "X-Msedge-Ref": [ "Ref A: 195B3CCB03964C91A2E307019CCAF8FF Ref B: CH1AA2020610017 Ref C: 2026-04-03T07:45:58Z", ], "X-Envoy-Upstream-Service-Time": ["94"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Correlation-Request-Id": [ "00b9e938-05fe-4cc6-bc31-bd2cf021e23c", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 03 Apr 2026 07:45:58 GMT", ], "Content-Length": ["107"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "e6351019-162e-4755-8111-22c1271ffc29", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a92ac0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", Us... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error: <fmt.wrapError \| 0xc000a92c60>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ml5zmp-22l25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ml5zmp/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ml5zmp-22l25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ml5zmp/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000b47860>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/a147bbb3-95ba-46a8-99a0-b2bba5e615c9", ], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "Expires": ["-1"], "Cache-Control": ["no-cache"], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T074559Z:00b9e938-05fe-4cc6-bc31-bd2cf021e23c", ], "X-Msedge-Ref": [ "Ref A: 195B3CCB03964C91A2E307019CCAF8FF Ref B: CH1AA2020610017 Ref C: 2026-04-03T07:45:58Z", ], "X-Envoy-Upstream-Service-Time": ["94"], "X-Ms-Failure-Cause": ["service"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "X-Ms-Correlation-Request-Id": [ "00b9e938-05fe-4cc6-bc31-bd2cf021e23c", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 03 Apr 2026 07:45:58 GMT", ], "Content-Length": ["107"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "e6351019-162e-4755-8111-22c1271ffc29", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a92ac0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", Us... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Interrupted by User 06:22:01 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:42]: Interrupted by User ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 06:22:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapError \| 0xc000e00fe0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-vqx9qs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-ksxbzz/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-vqx9qs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-ksxbzz/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012d3920>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/c1c5a103-be87-4c98-bf3c-144a505f8070", ], "X-Ms-Correlation-Request-Id": [ "811da485-3f04-4e6d-9e23-ab5e10af2a66", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["120"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T070704Z:811da485-3f04-4e6d-9e23-ab5e10af2a66", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 8278121BA9184CB79EDF1D3D2DC56AFF Ref B: CHI261140514036 Ref C: 2026-04-03T07:07:03Z", ], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Expires": ["-1"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "d9ee2ff2-a6a5-4e83-abb9-fd2d8ac19de7", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "Date": [ "Fri, 03 Apr 2026 07:07:03 GMT", ], "Pragma": ["no-cache"], }, Body: <exported.nopClosingBytesReader \| 0xc000e00f20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapError \| 0xc000e00fe0>: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-vqx9qs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-ksxbzz/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-vqx9qs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-ksxbzz/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012d3920>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Failure-Cause": ["service"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/switzerlandnorth/c1c5a103-be87-4c98-bf3c-144a505f8070", ], "X-Ms-Correlation-Request-Id": [ "811da485-3f04-4e6d-9e23-ab5e10af2a66", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["120"], "X-Ms-Routing-Request-Id": [ "SWITZERLANDNORTH:20260403T070704Z:811da485-3f04-4e6d-9e23-ab5e10af2a66", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 8278121BA9184CB79EDF1D3D2DC56AFF Ref B: CHI261140514036 Ref C: 2026-04-03T07:07:03Z", ], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"], "Expires": ["-1"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "d9ee2ff2-a6a5-4e83-abb9-fd2d8ac19de7", ], "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"], "Date": [ "Fri, 03 Apr 2026 07:07:03 GMT", ], "Pragma": ["no-cache"], }, Body: <exported.nopClosingBytesReader \| 0xc000e00f20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:22:01 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc00138aab0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4f544m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4f544m, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4f544m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4f544m, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00074af00>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4f544m", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc00138aab0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4f544m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4f544m, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-4f544m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4f544m, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00074af00>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-4f544m", }, <context.deadlineExceededError>{}, ], } occurred
06:20:10	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e, unknown	failed	n/a	11	e2e (10) timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc0009c8690>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-r4tzb7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-r4tzb7, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-r4tzb7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-r4tzb7, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000ade800>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-r4tzb7", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error: <fmt.wrapErrors \| 0xc0009c8690>: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-r4tzb7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-r4tzb7, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-r4tzb7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-r4tzb7, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000ade800>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-r4tzb7", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc0012ba9f0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-v9zh48" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-v9zh48, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-v9zh48\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-v9zh48, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006e9de0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-v9zh48", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error: <fmt.wrapErrors \| 0xc0012ba9f0>: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-v9zh48" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-v9zh48, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-v9zh48\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-v9zh48, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006e9de0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-v9zh48", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc0013802d0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6ksz9b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-6ksz9b, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6ksz9b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-6ksz9b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0002c2510>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-6ksz9b", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapErrors \| 0xc0013802d0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6ksz9b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-6ksz9b, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6ksz9b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-6ksz9b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0002c2510>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-6ksz9b", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error: <fmt.wrapErrors \| 0xc000c5f350>: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-gpfwx5" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-gpfwx5, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gpfwx5\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-gpfwx5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00065e970>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-gpfwx5", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error: <fmt.wrapErrors \| 0xc000c5f350>: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-gpfwx5" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-gpfwx5, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gpfwx5\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-gpfwx5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00065e970>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-gpfwx5", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc000f48d80>: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-c26ntd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-c26ntd, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-c26ntd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-c26ntd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0005a50c0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-c26ntd", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error: <fmt.wrapErrors \| 0xc000f48d80>: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-c26ntd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-c26ntd, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-c26ntd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-c26ntd, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0005a50c0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-c26ntd", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0008ea660>: failed waiting for hcpCluster="cluster-zstream-4-19-7gtk58" in resourcegroup="rg-zstream-upgrade-4-19-sgw4tc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-7gtk58 in resource group rg-zstream-upgrade-4-19-sgw4tc, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-7gtk58\" in resourcegroup=\"rg-zstream-upgrade-4-19-sgw4tc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-7gtk58 in resource group rg-zstream-upgrade-4-19-sgw4tc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0014f3a80>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-7gtk58 in resource group rg-zstream-upgrade-4-19-sgw4tc", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error: <fmt.wrapErrors \| 0xc0008ea660>: failed waiting for hcpCluster="cluster-zstream-4-19-7gtk58" in resourcegroup="rg-zstream-upgrade-4-19-sgw4tc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-7gtk58 in resource group rg-zstream-upgrade-4-19-sgw4tc, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-7gtk58\" in resourcegroup=\"rg-zstream-upgrade-4-19-sgw4tc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-7gtk58 in resource group rg-zstream-upgrade-4-19-sgw4tc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0014f3a80>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-7gtk58 in resource group rg-zstream-upgrade-4-19-sgw4tc", }, <context.deadlineExceededError>{}, ], } occurred timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc00090f260>: failed waiting for hcpCluster="admin-cred-lifecycle-z2bmsn" in resourcegroup="admin-credential-lifecycle-test-27w7lw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-z2bmsn in resource group admin-credential-lifecycle-test-27w7lw, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-z2bmsn\" in resourcegroup=\"admin-credential-lifecycle-test-27w7lw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-z2bmsn in resource group admin-credential-lifecycle-test-27w7lw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001390e00>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-z2bmsn in resource group admin-credential-lifecycle-test-27w7lw", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error: <fmt.wrapErrors \| 0xc00090f260>: failed waiting for hcpCluster="admin-cred-lifecycle-z2bmsn" in resourcegroup="admin-credential-lifecycle-test-27w7lw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-z2bmsn in resource group admin-credential-lifecycle-test-27w7lw, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-z2bmsn\" in resourcegroup=\"admin-credential-lifecycle-test-27w7lw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-z2bmsn in resource group admin-credential-lifecycle-test-27w7lw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001390e00>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-z2bmsn in resource group admin-credential-lifecycle-test-27w7lw", }, <context.deadlineExceededError>{}, ], } occurred Interrupted by User 06:20:10 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User Deserializaion Error: no output from command 06:20:10 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775200707 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 STEP: creating resource group @ 04/03/26 07:18:32.861 "ts"="2026-04-03 07:18:32.861166" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" STEP: creating cluster parameters at control plane version @ 04/03/26 07:18:36.582 STEP: creating customer resources @ 04/03/26 07:18:36.663 STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 07:19:30.275 "ts"="2026-04-03 07:19:30.276042" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-n644j9" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 07:32:30.616 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 07:43:09.56 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 07:50:07.133 STEP: capturing node release images before upgrade @ 04/03/26 07:50:07.268 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 07:50:07.316 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 07:51:18.942 STEP: verifying node pool GET still reflects the new version @ 04/03/26 08:21:21.401 "ts"="2026-04-03 08:21:21.916429" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 08:21:21.916513" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-n644j9-tzw4ms" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "ts"="2026-04-03 08:21:22.021919" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-n644j9-tzw4ms" "resourceGroup"="" "ts"="2026-04-03 08:21:22.381506" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 08:21:22.381609" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 08:21:22.381674" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 08:21:22.383217" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-5" "ts"="2026-04-03 08:21:22.383231" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "ts"="2026-04-03 08:21:22.482394" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "deployment"="pooledNonMsiScopedAssignments-ohfyjjvkpxkig" "ts"="2026-04-03 08:21:22.529060" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-5" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 08:21:22.548926" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "deployment"="customer-infra-np-version-upgrade-cluster-n644j9-tzw4ms" "ts"="2026-04-03 08:21:22.689884" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-5" "deployment"="msi-deployment-5" "ts"="2026-04-03 08:21:22.822113" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 08:21:22.822251" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 08:21:22.822398" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" Command Error: signal: killed Deserializaion Error: no output from command Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP ============================================================ Random Seed: 1775200707 - will randomize all specs Will run 1 of 1 specs ------------------------------ Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible] /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231 STEP: creating resource group @ 04/03/26 07:18:32.861 "ts"="2026-04-03 07:18:32.861166" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" STEP: creating cluster parameters at control plane version @ 04/03/26 07:18:36.582 STEP: creating customer resources @ 04/03/26 07:18:36.663 STEP: creating the HCP cluster with version 4.21.7 @ 04/03/26 07:19:30.275 "ts"="2026-04-03 07:19:30.276042" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-n644j9" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/03/26 07:32:30.616 STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/03/26 07:43:09.56 STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/03/26 07:50:07.133 STEP: capturing node release images before upgrade @ 04/03/26 07:50:07.268 STEP: triggering nodepool upgrade to version 4.21.7 @ 04/03/26 07:50:07.316 STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/03/26 07:51:18.942 STEP: verifying node pool GET still reflects the new version @ 04/03/26 08:21:21.401 "ts"="2026-04-03 08:21:21.916429" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-04-03 08:21:21.916513" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-n644j9-tzw4ms" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "ts"="2026-04-03 08:21:22.021919" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-n644j9-tzw4ms" "resourceGroup"="" "ts"="2026-04-03 08:21:22.381506" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 08:21:22.381609" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz" "ts"="2026-04-03 08:21:22.381674" "level"=0 "msg"="collecting debug info" "ts"="2026-04-03 08:21:22.383217" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-5" "ts"="2026-04-03 08:21:22.383231" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "ts"="2026-04-03 08:21:22.482394" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "deployment"="pooledNonMsiScopedAssignments-ohfyjjvkpxkig" "ts"="2026-04-03 08:21:22.529060" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-5" "deployment"="pooledMsiScopedAssignments" "ts"="2026-04-03 08:21:22.548926" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" "deployment"="customer-infra-np-version-upgrade-cluster-n644j9-tzw4ms" "ts"="2026-04-03 08:21:22.689884" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-5" "deployment"="msi-deployment-5" "ts"="2026-04-03 08:21:22.822113" "level"=0 "msg"="finished collecting debug info" "ts"="2026-04-03 08:21:22.822251" "level"=0 "msg"="deleting created resources" "ts"="2026-04-03 08:21:22.822398" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-n644j9-67j8pv" Command Error: signal: killed Deserializaion Error: no output from command timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds 06:20:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc00106d830>: failed waiting for hcpCluster="cluster-stable-4-20-bqjj6q" in resourcegroup="rg-stable-4-20-5xlww9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-bqjj6q in resource group rg-stable-4-20-5xlww9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-bqjj6q\" in resourcegroup=\"rg-stable-4-20-5xlww9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-bqjj6q in resource group rg-stable-4-20-5xlww9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001241df0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-bqjj6q in resource group rg-stable-4-20-5xlww9", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error: <fmt.wrapErrors \| 0xc00106d830>: failed waiting for hcpCluster="cluster-stable-4-20-bqjj6q" in resourcegroup="rg-stable-4-20-5xlww9" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-bqjj6q in resource group rg-stable-4-20-5xlww9, error: context deadline exceeded { msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-bqjj6q\" in resourcegroup=\"rg-stable-4-20-5xlww9\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-bqjj6q in resource group rg-stable-4-20-5xlww9, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001241df0>{ s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-bqjj6q in resource group rg-stable-4-20-5xlww9", }, <context.deadlineExceededError>{}, ], } occurred unknown (1) Process did not finish before 2h0m0s timeout 06:20:10 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-03T08:22:36Z"}
02:00:44	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) Interrupted by User 02:00:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User