CIHealth
<
Single Day: Apr 7▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e, unknown failed n/a 8
e2e (7)
Deserializaion Error: no output from command
04:05:55 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775539380 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 05:23:01.232
  "ts"="2026-04-07 05:23:01.232866" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-tczjgc-tp2rpx"
    STEP: creating cluster parameters at control plane version @ 04/07/26 05:23:05.133
    STEP: creating customer resources @ 04/07/26 05:23:05.225
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 05:23:50.322
  "ts"="2026-04-07 05:23:50.322296" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-tczjgc" "resourceGroup"="rg-np-version-upgrade-tczjgc-tp2rpx"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 05:34:54.824
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775539380 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 05:23:01.232
  "ts"="2026-04-07 05:23:01.232866" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-tczjgc-tp2rpx"
    STEP: creating cluster parameters at control plane version @ 04/07/26 05:23:05.133
    STEP: creating customer resources @ 04/07/26 05:23:05.225
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 05:23:50.322
  "ts"="2026-04-07 05:23:50.322296" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-tczjgc" "resourceGroup"="rg-np-version-upgrade-tczjgc-tp2rpx"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 05:34:54.824
Command Error: signal: killed
Deserializaion Error: no output from command
Interrupted by User
04:05:55 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44]: Interrupted by User
Interrupted by User
04:05:55 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:45]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:45]: Interrupted by User
Deserializaion Error: no output from command
04:05:55 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775539371 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
    STEP: creating resource group @ 04/07/26 05:23:00.777
  "ts"="2026-04-07 05:23:00.777205" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-9c578f-xr8n8x"
    STEP: creating cluster parameters at control plane version @ 04/07/26 05:23:04.384
    STEP: creating customer resources @ 04/07/26 05:23:04.465
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 05:23:51.414
  "ts"="2026-04-07 05:23:51.414251" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-9c578f" "resourceGroup"="rg-np-version-upgrade-9c578f-xr8n8x"
    STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 05:34:55.888
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/07/26 05:45:52.841
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/07/26 05:48:21.331
    STEP: capturing node release images before upgrade @ 04/07/26 05:48:21.463
    STEP: triggering nodepool upgrade to version 4.21.7 @ 04/07/26 05:48:21.503
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/07/26 05:49:13.715
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775539371 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
    STEP: creating resource group @ 04/07/26 05:23:00.777
  "ts"="2026-04-07 05:23:00.777205" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-9c578f-xr8n8x"
    STEP: creating cluster parameters at control plane version @ 04/07/26 05:23:04.384
    STEP: creating customer resources @ 04/07/26 05:23:04.465
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 05:23:51.414
  "ts"="2026-04-07 05:23:51.414251" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-9c578f" "resourceGroup"="rg-np-version-upgrade-9c578f-xr8n8x"
    STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 05:34:55.888
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/07/26 05:45:52.841
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/07/26 05:48:21.331
    STEP: capturing node release images before upgrade @ 04/07/26 05:48:21.463
    STEP: triggering nodepool upgrade to version 4.21.7 @ 04/07/26 05:48:21.503
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/07/26 05:49:13.715
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
04:05:55 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
04/07/26 05:38:04.367
    STEP: attempting to update nodepool version to higher than cluster version @ 04/07/26 05:48:19.896
  "ts"="2026-04-07 05:48:20.460472" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-l57kvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n    \"target\": \"properties.version.id\"\n  }\n}\n--------------------------------------------------------------------------------\n"
    STEP: attempting to update immutable platform profile fields @ 04/07/26 05:48:20.46
  "ts"="2026-04-07 05:48:21.151220" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-l57kvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"4 errors occurred\",\n    \"details\": [\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.vmSize\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.osDisk\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.availabilityZone\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
  "ts"="2026-04-07 05:48:21.271777" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 05:48:21.271880" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-rv6wgd" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 05:48:21.346235" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-rv6wgd" "resourceGroup"=""
  "ts"="2026-04-07 05:48:22.942477" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
  "ts"="2026-04-07 05:48:22.942620" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
  "ts"="2026-04-07 05:48:22.942678" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 05:48:22.944144" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-13"
  "ts"="2026-04-07 05:48:22.944161" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 05:48:23.204714" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-l57kvq" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy"
  "ts"="2026-04-07 05:48:23.305309" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-l57kvq" "deployment"="customer-infra-negative-tests-cluster-rv6wgd"
  "ts"="2026-04-07 05:48:23.320326" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-13" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 05:48:23.825441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-13" "deployment"="msi-deployment-13"
  "ts"="2026-04-07 05:48:24.341336" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 05:48:24.341454" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 05:48:24.341626" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 06:07:46.163294" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 06:07:46.163320" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-negative-tests-l57kvq"
Command Error: signal: killed
Deserializaion Error: no output from command
04/07/26 05:38:04.367
    STEP: attempting to update nodepool version to higher than cluster version @ 04/07/26 05:48:19.896
  "ts"="2026-04-07 05:48:20.460472" "msg"="invalid node pool version update" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-l57kvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Invalid value: \\\"4.21.0\\\": invalid node pool version 4.21.0: cannot exceed control plane version 4.20.16\",\n    \"target\": \"properties.version.id\"\n  }\n}\n--------------------------------------------------------------------------------\n"
    STEP: attempting to update immutable platform profile fields @ 04/07/26 05:48:20.46
  "ts"="2026-04-07 05:48:21.151220" "msg"="updating immutable fields" "error"="PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-l57kvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"4 errors occurred\",\n    \"details\": [\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.vmSize\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.osDisk\"\n      },\n      {\n        \"code\": \"InvalidRequestContent\",\n        \"message\": \"Forbidden: field is immutable\",\n        \"target\": \"properties.platform.availabilityZone\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
  "ts"="2026-04-07 05:48:21.271777" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 05:48:21.271880" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-negative-tests-cluster-rv6wgd" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 05:48:21.346235" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-negative-tests-cluster-rv6wgd" "resourceGroup"=""
  "ts"="2026-04-07 05:48:22.942477" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
  "ts"="2026-04-07 05:48:22.942620" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-92ac8e02d20de6cf37d354d7e01d22da1b3bf7e3835a3ed2f6f6742a237a853f.yaml.gz"
  "ts"="2026-04-07 05:48:22.942678" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 05:48:22.944144" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-13"
  "ts"="2026-04-07 05:48:22.944161" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 05:48:23.204714" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-l57kvq" "deployment"="pooledNonMsiScopedAssignments-oeayaalhj7wjy"
  "ts"="2026-04-07 05:48:23.305309" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-negative-tests-l57kvq" "deployment"="customer-infra-negative-tests-cluster-rv6wgd"
  "ts"="2026-04-07 05:48:23.320326" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-13" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 05:48:23.825441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-13" "deployment"="msi-deployment-13"
  "ts"="2026-04-07 05:48:24.341336" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 05:48:24.341454" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 05:48:24.341626" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 06:07:46.163294" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-negative-tests-l57kvq"
  "ts"="2026-04-07 06:07:46.163320" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-negative-tests-l57kvq"
Command Error: signal: killed
Deserializaion Error: no output from command
Interrupted by User
04:05:55 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:51]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:51]: Interrupted by User
Deserializaion Error: no output from command
04:05:55 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775539368 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
    STEP: creating a resource group @ 04/07/26 05:23:00.767
  "ts"="2026-04-07 05:23:00.767797" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-mbhzml"
    STEP: creating cluster parameters @ 04/07/26 05:23:05.374
    STEP: creating customer resources @ 04/07/26 05:23:05.374
    STEP: creating the HCP cluster @ 04/07/26 05:24:00.793
  "ts"="2026-04-07 05:24:00.794080" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-mbhzml"
    STEP: getting credentials @ 04/07/26 05:36:13.695
    STEP: creating the node pool with initial labels and taints @ 04/07/26 05:37:48.46
    STEP: waiting for nodes to be ready @ 04/07/26 05:49:58.944
    STEP: verifying initial labels are present on nodes @ 04/07/26 05:49:59.105
    STEP: verifying initial taints are present on nodes @ 04/07/26 05:49:59.128
    STEP: updating nodepool with new taints and scaling up @ 04/07/26 05:49:59.128
    STEP: verifying nodes are scaled to the expected count @ 04/07/26 05:59:46.317
    STEP: verifying new taint is present on newly created node @ 04/07/26 05:59:46.477
    STEP: updating nodepool with a new label and scaling up @ 04/07/26 05:59:46.499
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775539368 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
    STEP: creating a resource group @ 04/07/26 05:23:00.767
  "ts"="2026-04-07 05:23:00.767797" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-mbhzml"
    STEP: creating cluster parameters @ 04/07/26 05:23:05.374
    STEP: creating customer resources @ 04/07/26 05:23:05.374
    STEP: creating the HCP cluster @ 04/07/26 05:24:00.793
  "ts"="2026-04-07 05:24:00.794080" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-mbhzml"
    STEP: getting credentials @ 04/07/26 05:36:13.695
    STEP: creating the node pool with initial labels and taints @ 04/07/26 05:37:48.46
    STEP: waiting for nodes to be ready @ 04/07/26 05:49:58.944
    STEP: verifying initial labels are present on nodes @ 04/07/26 05:49:59.105
    STEP: verifying initial taints are present on nodes @ 04/07/26 05:49:59.128
    STEP: updating nodepool with new taints and scaling up @ 04/07/26 05:49:59.128
    STEP: verifying nodes are scaled to the expected count @ 04/07/26 05:59:46.317
    STEP: verifying new taint is present on newly created node @ 04/07/26 05:59:46.477
    STEP: updating nodepool with a new label and scaling up @ 04/07/26 05:59:46.499
Command Error: signal: killed
Deserializaion Error: no output from command
unknown (1)
Process did not finish before 2h0m0s timeout
04:05:55 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit
Show raw failure
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-07T06:08:14Z"}
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e, unknown failed n/a 4
e2e (3)
Deserializaion Error: no output from command
04:05:16 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
l run any cleanup and reporting nodes but will skip all remaining specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Authorized CIDRs Connectivity should allow API access only from authorized VM IP (Spec Runtime: 1h29m59.887s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:45
        In [It] (Node Runtime: 1h29m59.887s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:45
          At [By Step] updating cluster to remove VM from authorized CIDRs (Step Runtime: 1m1.446s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:392

          Spec Goroutine
          goroutine 123 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x7f1a4e242a60, 0xc000f141e0}, 0x81dda58?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cebf60, {0x7f1a4e242a60, 0xc000f141e0}, 0xc0016d0cf0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
          > github.com/Azure/ARO-HCP/test/e2e.init.func5.1.1({0x7f1a4e242a60, 0xc000f141e0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:417
                | Expect(err).NotTo(HaveOccurred())
                | 
                > _, err = poller.PollUntilDone(ctx, nil)
                | Expect(err).NotTo(HaveOccurred())
                | 
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc8d40?, 0xc000f141e0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 121
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:418 @ 04/07/26 05:37:42.895
  "ts"="2026-04-07 05:37:42.895340" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 05:37:42.895411" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cidr-connectivity-test-fkzk9c" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
  "ts"="2026-04-07 05:37:43.640257" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cidr-connectivity-test-fkzk9c" "resourceGroup"=""
  "ts"="2026-04-07 05:37:45.800087" "level"=0 "msg"="Dumping deployment operations." "deployment"="test-vm" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
  "ts"="2026-04-07 05:37:46.577328" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz"
  "ts"="2026-04-07 05:37:46.577436" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz"
  "ts"="2026-04-07 05:37:46.577508" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 05:37:46.578979" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-80"
  "ts"="2026-04-07 05:37:46.579001" "level"=0 "msg"="collecting deployments" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
  "ts"="2026-04-07 05:37:46.885972" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-80" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 05:37:47.023444" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-80" "deployment"="msi-deployment-80"
  "ts"="2026-04-07 05:37:47.068942" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-mbl5kf" "deployment"="test-vm"
  "ts"="2026-04-07 05:37:47.818656" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-mbl5kf" "deployment"="pooledNonMsiScopedAssignments-wihoigd7cpguy"
  "ts"="2026-04-07 05:37:48.556118" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-mbl5kf" "deployment"="customer-infra-cidr-connectivity-test-fkzk9c"
  "ts"="2026-04-07 05:37:49.295808" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 05:37:49.295900" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 05:37:49.296042" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
Command Error: signal: killed
Deserializaion Error: no output from command
l run any cleanup and reporting nodes but will skip all remaining specs.  Interrupt again to skip cleanup.
    Here's a current progress report:
      Authorized CIDRs Connectivity should allow API access only from authorized VM IP (Spec Runtime: 1h29m59.887s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:45
        In [It] (Node Runtime: 1h29m59.887s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:45
          At [By Step] updating cluster to remove VM from authorized CIDRs (Step Runtime: 1m1.446s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:392

          Spec Goroutine
          goroutine 123 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x7f1a4e242a60, 0xc000f141e0}, 0x81dda58?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cebf60, {0x7f1a4e242a60, 0xc000f141e0}, 0xc0016d0cf0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
          > github.com/Azure/ARO-HCP/test/e2e.init.func5.1.1({0x7f1a4e242a60, 0xc000f141e0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:417
                | Expect(err).NotTo(HaveOccurred())
                | 
                > _, err = poller.PollUntilDone(ctx, nil)
                | Expect(err).NotTo(HaveOccurred())
                | 
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8cc8d40?, 0xc000f141e0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 121
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:418 @ 04/07/26 05:37:42.895
  "ts"="2026-04-07 05:37:42.895340" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 05:37:42.895411" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cidr-connectivity-test-fkzk9c" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
  "ts"="2026-04-07 05:37:43.640257" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cidr-connectivity-test-fkzk9c" "resourceGroup"=""
  "ts"="2026-04-07 05:37:45.800087" "level"=0 "msg"="Dumping deployment operations." "deployment"="test-vm" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
  "ts"="2026-04-07 05:37:46.577328" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz"
  "ts"="2026-04-07 05:37:46.577436" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-5585d0232685e8d5bbce4642dbb894d73a2597e55ff59b2a6006f2460a217da1.yaml.gz"
  "ts"="2026-04-07 05:37:46.577508" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 05:37:46.578979" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-80"
  "ts"="2026-04-07 05:37:46.579001" "level"=0 "msg"="collecting deployments" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
  "ts"="2026-04-07 05:37:46.885972" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-80" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 05:37:47.023444" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-80" "deployment"="msi-deployment-80"
  "ts"="2026-04-07 05:37:47.068942" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-mbl5kf" "deployment"="test-vm"
  "ts"="2026-04-07 05:37:47.818656" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-mbl5kf" "deployment"="pooledNonMsiScopedAssignments-wihoigd7cpguy"
  "ts"="2026-04-07 05:37:48.556118" "level"=0 "msg"="collecting operations" "resourceGroup"="e2e-cidr-connectivity-mbl5kf" "deployment"="customer-infra-cidr-connectivity-test-fkzk9c"
  "ts"="2026-04-07 05:37:49.295808" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 05:37:49.295900" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 05:37:49.296042" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="e2e-cidr-connectivity-mbl5kf"
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
04:05:16 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
rnetes.io/os"="linux" "kubernetes.io/hostname"="p2h5q7b0k0o4w8k-np-1-lh727-jwtwb" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/zone"="1" "beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="brazilsouth" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/region"="brazilsouth" "hypershift.openshift.io/nodePool"="p2h5q7b0k0o4w8k-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "failure-domain.beta.kubernetes.io/zone"="1" "hypershift.openshift.io/managed"="true" "key1"="value1" "kubernetes.io/arch"="amd64"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}]
  "ts"="2026-04-07 05:58:53.380998" "level"=0 "msg"="Node" "name"="p2h5q7b0k0o4w8k-np-1-lh727-mn9jj" "labels"={"beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "node-role.kubernetes.io/worker"="" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/hostname"="p2h5q7b0k0o4w8k-np-1-lh727-mn9jj" "node.openshift.io/os_id"="rhel" "hypershift.openshift.io/nodePool"="p2h5q7b0k0o4w8k-np-1" "key1"="value1" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="brazilsouth" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="brazilsouth" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/arch"="amd64"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}]
  "ts"="2026-04-07 05:58:53.381045" "level"=0 "msg"="Node" "name"="p2h5q7b0k0o4w8k-np-1-lh727-nvdvx" "labels"={"beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="brazilsouth" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "key2"="value2" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/nodePool"="p2h5q7b0k0o4w8k-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/hostname"="p2h5q7b0k0o4w8k-np-1-lh727-nvdvx" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "kubernetes.io/arch"="amd64" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/region"="brazilsouth" "topology.kubernetes.io/zone"="0"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}]
    STEP: verifying original labels persist on 3 nodes @ 04/07/26 05:58:53.381
    STEP: verifying original taints persist on 2 nodes @ 04/07/26 05:58:53.381
  "ts"="2026-04-07 05:58:53.381321" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 05:58:53.381397" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-9hptlr" "resourceGroup"="rg-np-labels-taints-fvhpb2"
  "ts"="2026-04-07 05:58:54.127914" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-9hptlr" "resourceGroup"=""
  "ts"="2026-04-07 05:58:55.905622" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 05:58:55.905719" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 05:58:55.905778" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 05:58:55.907347" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-74"
  "ts"="2026-04-07 05:58:55.907358" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-fvhpb2"
  "ts"="2026-04-07 05:58:56.059340" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-74" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 05:58:56.342319" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-74" "deployment"="msi-deployment-74"
  "ts"="2026-04-07 05:58:56.377890" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fvhpb2" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
  "ts"="2026-04-07 05:58:57.390414" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fvhpb2" "deployment"="customer-infra-cluster-np-labels-taints-9hptlr"
  "ts"="2026-04-07 05:58:58.337339" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 05:58:58.337474" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 05:58:58.337604" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-fvhpb2"
Command Error: signal: killed
Deserializaion Error: no output from command
rnetes.io/os"="linux" "kubernetes.io/hostname"="p2h5q7b0k0o4w8k-np-1-lh727-jwtwb" "node.openshift.io/os_id"="rhel" "topology.kubernetes.io/zone"="1" "beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="brazilsouth" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/region"="brazilsouth" "hypershift.openshift.io/nodePool"="p2h5q7b0k0o4w8k-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "failure-domain.beta.kubernetes.io/zone"="1" "hypershift.openshift.io/managed"="true" "key1"="value1" "kubernetes.io/arch"="amd64"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}]
  "ts"="2026-04-07 05:58:53.380998" "level"=0 "msg"="Node" "name"="p2h5q7b0k0o4w8k-np-1-lh727-mn9jj" "labels"={"beta.kubernetes.io/instance-type"="Standard_D4s_v3" "hypershift.openshift.io/managed"="true" "kubernetes.io/arch"="amd64" "node-role.kubernetes.io/worker"="" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/hostname"="p2h5q7b0k0o4w8k-np-1-lh727-mn9jj" "node.openshift.io/os_id"="rhel" "hypershift.openshift.io/nodePool"="p2h5q7b0k0o4w8k-np-1" "key1"="value1" "topology.disk.csi.azure.com/zone"="" "topology.kubernetes.io/region"="brazilsouth" "topology.kubernetes.io/zone"="0" "beta.kubernetes.io/os"="linux" "failure-domain.beta.kubernetes.io/region"="brazilsouth" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/arch"="amd64"} "taints"=[{"key"="key1" "value"="value1" "effect"="NoSchedule"}]
  "ts"="2026-04-07 05:58:53.381045" "level"=0 "msg"="Node" "name"="p2h5q7b0k0o4w8k-np-1-lh727-nvdvx" "labels"={"beta.kubernetes.io/arch"="amd64" "failure-domain.beta.kubernetes.io/region"="brazilsouth" "failure-domain.beta.kubernetes.io/zone"="0" "hypershift.openshift.io/managed"="true" "key2"="value2" "node.openshift.io/os_id"="rhel" "beta.kubernetes.io/instance-type"="Standard_D4s_v3" "beta.kubernetes.io/os"="linux" "hypershift.openshift.io/nodePool"="p2h5q7b0k0o4w8k-np-1" "hypershift.openshift.io/nodepool-globalps-enabled"="true" "kubernetes.io/hostname"="p2h5q7b0k0o4w8k-np-1-lh727-nvdvx" "node-role.kubernetes.io/worker"="" "topology.disk.csi.azure.com/zone"="" "kubernetes.io/arch"="amd64" "kubernetes.io/os"="linux" "node.kubernetes.io/instance-type"="Standard_D4s_v3" "topology.kubernetes.io/region"="brazilsouth" "topology.kubernetes.io/zone"="0"} "taints"=[{"key"="key2" "value"="value2" "effect"="PreferNoSchedule"}]
    STEP: verifying original labels persist on 3 nodes @ 04/07/26 05:58:53.381
    STEP: verifying original taints persist on 2 nodes @ 04/07/26 05:58:53.381
  "ts"="2026-04-07 05:58:53.381321" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 05:58:53.381397" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-9hptlr" "resourceGroup"="rg-np-labels-taints-fvhpb2"
  "ts"="2026-04-07 05:58:54.127914" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-9hptlr" "resourceGroup"=""
  "ts"="2026-04-07 05:58:55.905622" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 05:58:55.905719" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 05:58:55.905778" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 05:58:55.907347" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-74"
  "ts"="2026-04-07 05:58:55.907358" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-fvhpb2"
  "ts"="2026-04-07 05:58:56.059340" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-74" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 05:58:56.342319" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-74" "deployment"="msi-deployment-74"
  "ts"="2026-04-07 05:58:56.377890" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fvhpb2" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
  "ts"="2026-04-07 05:58:57.390414" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-fvhpb2" "deployment"="customer-infra-cluster-np-labels-taints-9hptlr"
  "ts"="2026-04-07 05:58:58.337339" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 05:58:58.337474" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 05:58:58.337604" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-fvhpb2"
Command Error: signal: killed
Deserializaion Error: no output from command
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
04:05:16 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154]: Unexpected error:
    <*fmt.wrapErrors | 0xc000958690>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-pmp787" in resourcegroup="rg-np-version-upgrade-pmp787-kwgkwl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001248460>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl",
            },
            <*fmt.wrapError | 0xc001104340>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154]: Unexpected error:
    <*fmt.wrapErrors | 0xc000958690>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-pmp787" in resourcegroup="rg-np-version-upgrade-pmp787-kwgkwl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001248460>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl",
            },
            <*fmt.wrapError | 0xc001104340>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
unknown (1)
Process did not finish before 2h0m0s timeout
04:05:16 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit
Show raw failure
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-07T06:07:36Z"}
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e, unknown failed n/a 4
e2e (3)
Interrupted by User
04:04:40 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Deserializaion Error: no output from command
04:04:40 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775538149 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 05:02:29.603
  "ts"="2026-04-07 05:02:29.603820" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
    STEP: creating cluster parameters at control plane version @ 04/07/26 05:02:33.781
    STEP: creating customer resources @ 04/07/26 05:02:33.862
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 05:03:26.183
  "ts"="2026-04-07 05:03:26.183582" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-9mxh7s" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 05:19:16.826
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154 @ 04/07/26 06:04:16.833
  "ts"="2026-04-07 06:04:16.834185" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 06:04:16.834270" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-9mxh7s-2vp6d7" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
  "ts"="2026-04-07 06:04:17.366829" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-9mxh7s-2vp6d7" "resourceGroup"=""
  "ts"="2026-04-07 06:04:18.997857" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 06:04:18.997955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 06:04:18.998020" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 06:04:18.999596" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-1"
  "ts"="2026-04-07 06:04:18.999607" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
  "ts"="2026-04-07 06:04:19.153005" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-1" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 06:04:19.305470" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-1" "deployment"="msi-deployment-1"
  "ts"="2026-04-07 06:04:19.951537" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k" "deployment"="pooledNonMsiScopedAssignments-pr77ts5rpnbqu"
  "ts"="2026-04-07 06:04:20.715265" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k" "deployment"="customer-infra-np-version-upgrade-cluster-9mxh7s-2vp6d7"
  "ts"="2026-04-07 06:04:21.424419" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 06:04:21.424586" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 06:04:21.424739" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775538149 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 05:02:29.603
  "ts"="2026-04-07 05:02:29.603820" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
    STEP: creating cluster parameters at control plane version @ 04/07/26 05:02:33.781
    STEP: creating customer resources @ 04/07/26 05:02:33.862
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 05:03:26.183
  "ts"="2026-04-07 05:03:26.183582" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-9mxh7s" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 05:19:16.826
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154 @ 04/07/26 06:04:16.833
  "ts"="2026-04-07 06:04:16.834185" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 06:04:16.834270" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-9mxh7s-2vp6d7" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
  "ts"="2026-04-07 06:04:17.366829" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-9mxh7s-2vp6d7" "resourceGroup"=""
  "ts"="2026-04-07 06:04:18.997857" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 06:04:18.997955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 06:04:18.998020" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 06:04:18.999596" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-1"
  "ts"="2026-04-07 06:04:18.999607" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
  "ts"="2026-04-07 06:04:19.153005" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-1" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 06:04:19.305470" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-1" "deployment"="msi-deployment-1"
  "ts"="2026-04-07 06:04:19.951537" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k" "deployment"="pooledNonMsiScopedAssignments-pr77ts5rpnbqu"
  "ts"="2026-04-07 06:04:20.715265" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k" "deployment"="customer-infra-np-version-upgrade-cluster-9mxh7s-2vp6d7"
  "ts"="2026-04-07 06:04:21.424419" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 06:04:21.424586" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 06:04:21.424739" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-9mxh7s-dwkt6k"
Command Error: signal: killed
Deserializaion Error: no output from command
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
04:04:40 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error:
    <*fmt.wrapError | 0xc000f2c9e0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ztlcvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ztlcvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00178b8c0>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Correlation-Request-Id": [
                        "9df649c9-418f-4940-bec8-8c0e548e3db3",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Expires": ["-1"],
                    "Date": [
                        "Tue, 07 Apr 2026 05:20:56 GMT",
                    ],
                    "Pragma": ["no-cache"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "b0981fa0-473c-423b-8940-929e8b4ed066",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["1246"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Routing-Request-Id": [
                        "WESTEUROPE:20260407T052056Z:9df649c9-418f-4940-bec8-8c0e548e3db3",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Cache-Control": ["no-cache"],
                    "Content-Length": ["107"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/95c613cd-bdd4-496d-928d-a3fc6f2b736e",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 9189BB39768F4C5AA7879357F88C2139 Ref B: CHI261140501031 Ref C: 2026-04-07T05:20:55Z",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000f2c920>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "m...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:181]: Unexpected error:
    <*fmt.wrapError | 0xc000f2c9e0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ztlcvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ztlcvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00178b8c0>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Correlation-Request-Id": [
                        "9df649c9-418f-4940-bec8-8c0e548e3db3",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Expires": ["-1"],
                    "Date": [
                        "Tue, 07 Apr 2026 05:20:56 GMT",
                    ],
                    "Pragma": ["no-cache"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "b0981fa0-473c-423b-8940-929e8b4ed066",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["1246"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Routing-Request-Id": [
                        "WESTEUROPE:20260407T052056Z:9df649c9-418f-4940-bec8-8c0e548e3db3",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Cache-Control": ["no-cache"],
                    "Content-Length": ["107"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/westeurope/95c613cd-bdd4-496d-928d-a3fc6f2b736e",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 9189BB39768F4C5AA7879357F88C2139 Ref B: CHI261140501031 Ref C: 2026-04-07T05:20:55Z",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000f2c920>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "m...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
unknown (1)
Process did not finish before 2h0m0s timeout
04:04:40 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit
Show raw failure
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-07T06:07:00Z"}
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 32
e2e (32)
Deserializaion Error: no output from command
00:23:16 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526666 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237
    STEP: creating resource group @ 04/07/26 01:51:11.527
  "ts"="2026-04-07 01:51:11.527298" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-hhmb8r-pnkvwg"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:51:16.274
    STEP: creating customer resources @ 04/07/26 01:51:16.408
    STEP: creating the HCP cluster with version 4.20.17 @ 04/07/26 01:52:26.096
  "ts"="2026-04-07 01:52:26.096671" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-hhmb8r" "resourceGroup"="rg-np-version-upgrade-hhmb8r-pnkvwg"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526666 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:237
    STEP: creating resource group @ 04/07/26 01:51:11.527
  "ts"="2026-04-07 01:51:11.527298" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-hhmb8r-pnkvwg"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:51:16.274
    STEP: creating customer resources @ 04/07/26 01:51:16.408
    STEP: creating the HCP cluster with version 4.20.17 @ 04/07/26 01:52:26.096
  "ts"="2026-04-07 01:52:26.096671" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-hhmb8r" "resourceGroup"="rg-np-version-upgrade-hhmb8r-pnkvwg"
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
00:23:16 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526682 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41
    STEP: creating a resource group @ 04/07/26 01:51:22.035
  "ts"="2026-04-07 01:51:22.035287" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-2qcm89"
    STEP: creating cluster parameters @ 04/07/26 01:51:26.178
    STEP: creating customer resources @ 04/07/26 01:51:26.178
    STEP: creating the HCP cluster @ 04/07/26 01:52:43.309
  "ts"="2026-04-07 01:52:43.309255" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-2qcm89"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526682 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should not be able to perform various invalid operations on cluster resources [PreLaunchSetup:None, Positivity:Negative, Importance:Medium, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:41
    STEP: creating a resource group @ 04/07/26 01:51:22.035
  "ts"="2026-04-07 01:51:22.035287" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-negative-tests-2qcm89"
    STEP: creating cluster parameters @ 04/07/26 01:51:26.178
    STEP: creating customer resources @ 04/07/26 01:51:26.178
    STEP: creating the HCP cluster @ 04/07/26 01:52:43.309
  "ts"="2026-04-07 01:52:43.309255" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="negative-tests-cluster" "resourceGroup"="rg-negative-tests-2qcm89"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:23:16 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014db050>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bqjn7p" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bqjn7p, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bqjn7p\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bqjn7p, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000614870>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bqjn7p",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014db050>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bqjn7p" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bqjn7p, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-bqjn7p\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bqjn7p, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000614870>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bqjn7p",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Deserializaion Error: no output from command
00:23:16 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526671 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to use workload identity via the cluster OIDC issuer URL [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169
    STEP: creating a resource group @ 04/07/26 01:51:11.583
  "ts"="2026-04-07 01:51:11.583892" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-2sj8tw"
    STEP: creating cluster parameters @ 04/07/26 01:51:16.234
    STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/07/26 01:51:16.234
    STEP: creating the HCP cluster @ 04/07/26 01:52:38.669
  "ts"="2026-04-07 01:52:38.669995" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-2sj8tw"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526671 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to use workload identity via the cluster OIDC issuer URL [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:169
    STEP: creating a resource group @ 04/07/26 01:51:11.583
  "ts"="2026-04-07 01:51:11.583892" "level"=0 "msg"="creating resource group" "resourceGroup"="oidc-wi-2sj8tw"
    STEP: creating cluster parameters @ 04/07/26 01:51:16.234
    STEP: creating customer resources (infrastructure and managed identities) for cluster @ 04/07/26 01:51:16.234
    STEP: creating the HCP cluster @ 04/07/26 01:52:38.669
  "ts"="2026-04-07 01:52:38.669995" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="oidc-wi-cluster" "resourceGroup"="oidc-wi-2sj8tw"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011aa540>: 
    failed to create HCP cluster ea-listw6gcm5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: failed waiting for cluster="ea-listw6gcm5" in resourcegroup="ea-list-4w9cgg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listw6gcm5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: failed waiting for cluster=\"ea-listw6gcm5\" in resourcegroup=\"ea-list-4w9cgg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f6a6b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg",
            },
            <*fmt.wrapErrors | 0xc0011aa510>{
                msg: "failed waiting for cluster=\"ea-listw6gcm5\" in resourcegroup=\"ea-list-4w9cgg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f6a6b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011aa540>: 
    failed to create HCP cluster ea-listw6gcm5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: failed waiting for cluster="ea-listw6gcm5" in resourcegroup="ea-list-4w9cgg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listw6gcm5, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: failed waiting for cluster=\"ea-listw6gcm5\" in resourcegroup=\"ea-list-4w9cgg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000f6a6b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg",
            },
            <*fmt.wrapErrors | 0xc0011aa510>{
                msg: "failed waiting for cluster=\"ea-listw6gcm5\" in resourcegroup=\"ea-list-4w9cgg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000f6a6b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listw6gcm5 in resource group ea-list-4w9cgg",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:23:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc00150e3c0>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-hqc5zn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hqc5zn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-hqc5zn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hqc5zn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009ad190>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hqc5zn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc00150e3c0>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-hqc5zn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hqc5zn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-hqc5zn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hqc5zn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009ad190>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-hqc5zn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc001802780>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-5m87gs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-5m87gs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e3b670>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs",
            },
            <*fmt.wrapErrors | 0xc001802750>{
                msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-5m87gs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e3b670>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc001802780>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-5m87gs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-5m87gs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e3b670>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs",
            },
            <*fmt.wrapErrors | 0xc001802750>{
                msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-5m87gs\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e3b670>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-5m87gs",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009ea5a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-gsft5q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gsft5q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009484b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q",
            },
            <*fmt.wrapErrors | 0xc0009ea570>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gsft5q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009484b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009ea5a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-gsft5q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gsft5q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009484b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q",
            },
            <*fmt.wrapErrors | 0xc0009ea570>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-gsft5q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009484b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-gsft5q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012d86f0>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-b8kv4h" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001247440>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h",
            },
            <*fmt.wrapError | 0xc0005187e0>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012d86f0>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-b8kv4h" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001247440>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h",
            },
            <*fmt.wrapError | 0xc0005187e0>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:23:16 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012d06c0>: 
    failed waiting for hcpCluster="cluster-stable-4-20-5875bd" in resourcegroup="rg-stable-4-20-sx464q" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5875bd in resource group rg-stable-4-20-sx464q, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-5875bd\" in resourcegroup=\"rg-stable-4-20-sx464q\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5875bd in resource group rg-stable-4-20-sx464q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001362380>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5875bd in resource group rg-stable-4-20-sx464q",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012d06c0>: 
    failed waiting for hcpCluster="cluster-stable-4-20-5875bd" in resourcegroup="rg-stable-4-20-sx464q" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5875bd in resource group rg-stable-4-20-sx464q, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-5875bd\" in resourcegroup=\"rg-stable-4-20-sx464q\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5875bd in resource group rg-stable-4-20-sx464q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001362380>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-5875bd in resource group rg-stable-4-20-sx464q",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapErrors | 0xc00092ca20>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-98hkw2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-98hkw2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e839a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2",
            },
            <*fmt.wrapErrors | 0xc00092c9f0>{
                msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-98hkw2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e839a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapErrors | 0xc00092ca20>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-98hkw2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-98hkw2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e839a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2",
            },
            <*fmt.wrapErrors | 0xc00092c9f0>{
                msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-98hkw2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e839a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-98hkw2",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014d1560>: 
    failed to create HCP cluster cluster-zstream-4-20-s2sjhd, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: failed waiting for cluster="cluster-zstream-4-20-s2sjhd" in resourcegroup="rg-zstream-upgrade-4-20-tjt8qk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-s2sjhd, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: failed waiting for cluster=\"cluster-zstream-4-20-s2sjhd\" in resourcegroup=\"rg-zstream-upgrade-4-20-tjt8qk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013b6770>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk",
            },
            <*fmt.wrapErrors | 0xc0014d1530>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-20-s2sjhd\" in resourcegroup=\"rg-zstream-upgrade-4-20-tjt8qk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013b6770>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014d1560>: 
    failed to create HCP cluster cluster-zstream-4-20-s2sjhd, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: failed waiting for cluster="cluster-zstream-4-20-s2sjhd" in resourcegroup="rg-zstream-upgrade-4-20-tjt8qk" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-s2sjhd, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: failed waiting for cluster=\"cluster-zstream-4-20-s2sjhd\" in resourcegroup=\"rg-zstream-upgrade-4-20-tjt8qk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013b6770>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk",
            },
            <*fmt.wrapErrors | 0xc0014d1530>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-20-s2sjhd\" in resourcegroup=\"rg-zstream-upgrade-4-20-tjt8qk\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013b6770>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-s2sjhd in resource group rg-zstream-upgrade-4-20-tjt8qk",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:123]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011acae0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dvhdz5, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dvhdz5" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dvhdz5, error: failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-dvhdz5\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014c5b20>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dvhdz5",
            },
            <*fmt.wrapError | 0xc001086580>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-dvhdz5\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:123]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011acae0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dvhdz5, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dvhdz5" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dvhdz5, error: failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-dvhdz5\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014c5b20>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dvhdz5",
            },
            <*fmt.wrapError | 0xc001086580>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-dvhdz5\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
Interrupted by User
00:23:16 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at the resource group level · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:34]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:34]: Interrupted by User
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc001193290>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-879vc5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-879vc5, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-879vc5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-879vc5, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001180090>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-879vc5",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:108]: Unexpected error:
    <*fmt.wrapErrors | 0xc001193290>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-879vc5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-879vc5, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-879vc5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-879vc5, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001180090>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-879vc5",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc000bdb0e0>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-5b92nh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-5b92nh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e905f0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh",
            },
            <*fmt.wrapErrors | 0xc000bdb0b0>{
                msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-5b92nh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e905f0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc000bdb0e0>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-5b92nh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ephemeral-disk, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-5b92nh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e905f0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh",
            },
            <*fmt.wrapErrors | 0xc000bdb0b0>{
                msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-5b92nh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e905f0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-5b92nh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Deserializaion Error: no output from command
00:23:16 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel
Show raw failure
amd64.s:1693 +0x1 fp=0xc000bf1fe8 sp=0xc000bf1fe0 pc=0x2dbecc1
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230 +0x1e5

goroutine 119 gp=0xc000bb48c0 m=nil [select]:
runtime.gopark(0xc000e602f0?, 0x2?, 0xa8?, 0xf8?, 0xc000e602e4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000e60170 sp=0xc000e60150 pc=0x2db650e
runtime.selectgo(0xc000e602f0, 0xc000e602e0, 0xc000b8d940?, 0x0, 0x35cdd3c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000e602b0 sp=0xc000e60170 pc=0x2d912f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8cb0988, 0xc0009b4460}, 0x81dda58?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000e60320 sp=0xc000e602b0 pc=0x35cd496
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cebf60, {0x8cb0988, 0xc0009b4460}, 0x0)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x7b2 fp=0xc000e60668 sp=0xc000e60320 pc=0x3abad52
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8cb09f8?, 0xc000340fc0?}, {{0x8cc8c68?, 0xc000638b80?}, 0xc00120a680?}, 0xc000e60b70, {0xc000ea4260, 0x1e}, {0xc000bc2440, 0x1b}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694 +0x405 fp=0xc000e608e0 sp=0xc000e60668 pc=0x5261e85
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0xc00127d658, 0x7}, ...}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346 +0x497 fp=0xc000e60c40 sp=0xc000e608e0 pc=0x5255fd7
github.com/Azure/ARO-HCP/test/e2e.init.func19.1.1({0x66?, 0x0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:115 +0xa8 fp=0xc000e60e88 sp=0xc000e60c40 pc=0x5b49488
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x7fd5e40ff460?, 0xc00089faa0?}, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc000e60ee0 sp=0xc000e60e88 pc=0x3581842
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x7fd5e40ff460, 0xc00089faa0}, {0x12a05f200, 0x4000000000000000, 0x3fb999999999999a, 0x19, 0xa7a358200}, 0xc000e61540)
	/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc000e60f58 sp=0xc000e60ee0 pc=0x357f8ee
github.com/Azure/ARO-HCP/test/e2e.init.func19.1({0x7fd5e40ff460, 0xc00089faa0}, {0x81bbd5c, 0x4}, {0x81c3d5c, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:108 +0xb38 fp=0xc000e617f0 sp=0xc000e60f58 pc=0x5b48f78
runtime.call64(0xc00089fb60, 0x85a90b8, 0x0, 0x0, 0x0, 0x30, 0xc000e61a78)
	/usr/lib/golang/src/runtime/asm_amd64.s:776 +0x43 fp=0xc000e61840 sp=0xc000e617f0 pc=0x2dbd363
runtime.reflectcall(0x70bc6c0?, 0x8be7210?, 0x8?, 0x81fbd86?, 0x0?, 0x12?, 0x70bc6c0?)
	<autogenerated>:1 +0x36 fp=0xc000e61880 sp=0xc000e61840 pc=0x2dc5ab6
reflect.Value.call({0x7309280?, 0x85a90b8?, 0x13?}, {0x81bbe44, 0x4}, {0xc000a1b2c0, 0x3, 0x3?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000e61de0 sp=0xc000e61880 pc=0x2df8a86
reflect.Value.Call({0x7309280?, 0x85a90b8?, 0x8d4f0b0?}, {0xc000a1b2c0?, 0x30?, 0x730f260?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000e61e58 sp=0xc000e61de0 pc=0x2df7b79
github.com/onsi/ginkgo/v2.invokeFunction({0x7309280, 0x85a90b8}, {0xc00089fb30, 0x3, 0x730f260?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/table_dsl.go:339 +0x2c5 fp=0xc000e61f18 sp=0xc000e61e58 pc=0x323c2c5
github.com/onsi/ginkgo/v2.generateTable.func2.1({0x8cc8d40?, 0xc00089faa0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/table_dsl.go:293 +0x105 fp=0xc000e61f88 sp=0xc000e61f18 pc=0x323bfc5
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000e61fe0 sp=0xc000e61f88 pc=0x322dabb
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e61fe8 sp=0xc000e61fe0 pc=0x2dbecc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 117
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

rax    0xca
rbx    0x0
rcx    0x2dc0ae3
rdx    0x0
rdi    0xc1ba5b0
rsi    0x80
rbp    0x7fff607392a0
rsp    0x7fff60739258
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc0000de008
r13    0x1
r14    0xc1b84c0
r15    0x3
rip    0x2dc0ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
amd64.s:1693 +0x1 fp=0xc000bf1fe8 sp=0xc000bf1fe0 pc=0x2dbecc1
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230 +0x1e5

goroutine 119 gp=0xc000bb48c0 m=nil [select]:
runtime.gopark(0xc000e602f0?, 0x2?, 0xa8?, 0xf8?, 0xc000e602e4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000e60170 sp=0xc000e60150 pc=0x2db650e
runtime.selectgo(0xc000e602f0, 0xc000e602e0, 0xc000b8d940?, 0x0, 0x35cdd3c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000e602b0 sp=0xc000e60170 pc=0x2d912f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8cb0988, 0xc0009b4460}, 0x81dda58?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000e60320 sp=0xc000e602b0 pc=0x35cd496
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8cebf60, {0x8cb0988, 0xc0009b4460}, 0x0)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x7b2 fp=0xc000e60668 sp=0xc000e60320 pc=0x3abad52
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8cb09f8?, 0xc000340fc0?}, {{0x8cc8c68?, 0xc000638b80?}, 0xc00120a680?}, 0xc000e60b70, {0xc000ea4260, 0x1e}, {0xc000bc2440, 0x1b}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694 +0x405 fp=0xc000e608e0 sp=0xc000e60668 pc=0x5261e85
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0xc00127d658, 0x7}, ...}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346 +0x497 fp=0xc000e60c40 sp=0xc000e608e0 pc=0x5255fd7
github.com/Azure/ARO-HCP/test/e2e.init.func19.1.1({0x66?, 0x0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:115 +0xa8 fp=0xc000e60e88 sp=0xc000e60c40 pc=0x5b49488
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x7fd5e40ff460?, 0xc00089faa0?}, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc000e60ee0 sp=0xc000e60e88 pc=0x3581842
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x7fd5e40ff460, 0xc00089faa0}, {0x12a05f200, 0x4000000000000000, 0x3fb999999999999a, 0x19, 0xa7a358200}, 0xc000e61540)
	/go/pkg/mod/k8s.io/apimachinery@v0.34.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc000e60f58 sp=0xc000e60ee0 pc=0x357f8ee
github.com/Azure/ARO-HCP/test/e2e.init.func19.1({0x7fd5e40ff460, 0xc00089faa0}, {0x81bbd5c, 0x4}, {0x81c3d5c, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:108 +0xb38 fp=0xc000e617f0 sp=0xc000e60f58 pc=0x5b48f78
runtime.call64(0xc00089fb60, 0x85a90b8, 0x0, 0x0, 0x0, 0x30, 0xc000e61a78)
	/usr/lib/golang/src/runtime/asm_amd64.s:776 +0x43 fp=0xc000e61840 sp=0xc000e617f0 pc=0x2dbd363
runtime.reflectcall(0x70bc6c0?, 0x8be7210?, 0x8?, 0x81fbd86?, 0x0?, 0x12?, 0x70bc6c0?)
	<autogenerated>:1 +0x36 fp=0xc000e61880 sp=0xc000e61840 pc=0x2dc5ab6
reflect.Value.call({0x7309280?, 0x85a90b8?, 0x13?}, {0x81bbe44, 0x4}, {0xc000a1b2c0, 0x3, 0x3?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000e61de0 sp=0xc000e61880 pc=0x2df8a86
reflect.Value.Call({0x7309280?, 0x85a90b8?, 0x8d4f0b0?}, {0xc000a1b2c0?, 0x30?, 0x730f260?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000e61e58 sp=0xc000e61de0 pc=0x2df7b79
github.com/onsi/ginkgo/v2.invokeFunction({0x7309280, 0x85a90b8}, {0xc00089fb30, 0x3, 0x730f260?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/table_dsl.go:339 +0x2c5 fp=0xc000e61f18 sp=0xc000e61e58 pc=0x323c2c5
github.com/onsi/ginkgo/v2.generateTable.func2.1({0x8cc8d40?, 0xc00089faa0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/table_dsl.go:293 +0x105 fp=0xc000e61f88 sp=0xc000e61f18 pc=0x323bfc5
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000e61fe0 sp=0xc000e61f88 pc=0x322dabb
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e61fe8 sp=0xc000e61fe0 pc=0x2dbecc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 117
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

rax    0xca
rbx    0x0
rcx    0x2dc0ae3
rdx    0x0
rdi    0xc1ba5b0
rsi    0x80
rbp    0x7fff607392a0
rsp    0x7fff60739258
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc0000de008
r13    0x1
r14    0xc1b84c0
r15    0x3
rip    0x2dc0ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapErrors | 0xc001214ba0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-jb2xfp, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jb2xfp" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-jb2xfp, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jb2xfp\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00058d540>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-jb2xfp",
            },
            <*fmt.wrapError | 0xc0008b85e0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jb2xfp\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124]: Unexpected error:
    <*fmt.wrapErrors | 0xc001214ba0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-jb2xfp, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jb2xfp" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-jb2xfp, error: failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jb2xfp\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00058d540>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-jb2xfp",
            },
            <*fmt.wrapError | 0xc0008b85e0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jb2xfp\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:23:16 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc00152e090>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-w2k2l9" in resourcegroup="admin-credential-lifecycle-test-xnk8kd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-w2k2l9 in resource group admin-credential-lifecycle-test-xnk8kd, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-w2k2l9\" in resourcegroup=\"admin-credential-lifecycle-test-xnk8kd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-w2k2l9 in resource group admin-credential-lifecycle-test-xnk8kd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000dd89c0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-w2k2l9 in resource group admin-credential-lifecycle-test-xnk8kd",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc00152e090>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-w2k2l9" in resourcegroup="admin-credential-lifecycle-test-xnk8kd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-w2k2l9 in resource group admin-credential-lifecycle-test-xnk8kd, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-w2k2l9\" in resourcegroup=\"admin-credential-lifecycle-test-xnk8kd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-w2k2l9 in resource group admin-credential-lifecycle-test-xnk8kd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000dd89c0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-w2k2l9 in resource group admin-credential-lifecycle-test-xnk8kd",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:23:16 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e62d50>: 
    failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-rrjgvj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rrjgvj, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rrjgvj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rrjgvj, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00047a540>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rrjgvj",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e62d50>: 
    failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-rrjgvj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rrjgvj, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-rrjgvj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rrjgvj, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00047a540>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-rrjgvj",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Deserializaion Error: no output from command
00:23:16 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526605 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
    STEP: creating resource group @ 04/07/26 01:50:05.987
  "ts"="2026-04-07 01:50:05.987526" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-cgm98g-zfqtjt"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:50:10.608
    STEP: creating customer resources @ 04/07/26 01:50:10.697
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:51:30.392
  "ts"="2026-04-07 01:51:30.392820" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-cgm98g" "resourceGroup"="rg-np-version-upgrade-cgm98g-zfqtjt"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526605 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231
    STEP: creating resource group @ 04/07/26 01:50:05.987
  "ts"="2026-04-07 01:50:05.987526" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-cgm98g-zfqtjt"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:50:10.608
    STEP: creating customer resources @ 04/07/26 01:50:10.697
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:51:30.392
  "ts"="2026-04-07 01:51:30.392820" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-cgm98g" "resourceGroup"="rg-np-version-upgrade-cgm98g-zfqtjt"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:23:16 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc00158ce10>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-dqsrr2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dqsrr2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dqsrr2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dqsrr2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a7e9c0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dqsrr2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc00158ce10>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-dqsrr2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dqsrr2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dqsrr2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dqsrr2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a7e9c0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dqsrr2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Deserializaion Error: no output from command
00:23:16 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526656 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 01:50:57.267
  "ts"="2026-04-07 01:50:57.267206" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-mn8s95-8qq489"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:51:01.017
    STEP: creating customer resources @ 04/07/26 01:51:01.099
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:52:22.852
  "ts"="2026-04-07 01:52:22.852166" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-mn8s95" "resourceGroup"="rg-np-version-upgrade-mn8s95-8qq489"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775526656 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 01:50:57.267
  "ts"="2026-04-07 01:50:57.267206" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-mn8s95-8qq489"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:51:01.017
    STEP: creating customer resources @ 04/07/26 01:51:01.099
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:52:22.852
  "ts"="2026-04-07 01:52:22.852166" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-mn8s95" "resourceGroup"="rg-np-version-upgrade-mn8s95-8qq489"
Command Error: signal: killed
Deserializaion Error: no output from command
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapErrors | 0xc000930930>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-2mpbcw, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2mpbcw" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-2mpbcw, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2mpbcw\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000995180>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-2mpbcw",
            },
            <*fmt.wrapError | 0xc001406c00>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2mpbcw\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapErrors | 0xc000930930>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-2mpbcw, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2mpbcw" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-2mpbcw, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2mpbcw\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000995180>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-2mpbcw",
            },
            <*fmt.wrapError | 0xc001406c00>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2mpbcw\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
Interrupted by User
00:23:16 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:23:16 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010923c0>: 
    failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-rn4rfw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-rn4rfw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-rn4rfw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-rn4rfw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a21460>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-rn4rfw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010923c0>: 
    failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-rn4rfw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-rn4rfw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-rn4rfw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-rn4rfw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a21460>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-rn4rfw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc00152a210>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5wwxjp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-5wwxjp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00098cbd0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp",
            },
            <*fmt.wrapErrors | 0xc00152a1e0>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-5wwxjp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00098cbd0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc00152a210>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5wwxjp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-5wwxjp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00098cbd0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp",
            },
            <*fmt.wrapErrors | 0xc00152a1e0>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-5wwxjp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00098cbd0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-5wwxjp",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009a8750>: 
    failed to create HCP cluster cluster-zstream-4-21-fdsksh, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: failed waiting for cluster="cluster-zstream-4-21-fdsksh" in resourcegroup="rg-zstream-upgrade-4-21-ssph6j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-fdsksh, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: failed waiting for cluster=\"cluster-zstream-4-21-fdsksh\" in resourcegroup=\"rg-zstream-upgrade-4-21-ssph6j\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012d10b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j",
            },
            <*fmt.wrapErrors | 0xc0009a8720>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-21-fdsksh\" in resourcegroup=\"rg-zstream-upgrade-4-21-ssph6j\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0012d10b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009a8750>: 
    failed to create HCP cluster cluster-zstream-4-21-fdsksh, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: failed waiting for cluster="cluster-zstream-4-21-fdsksh" in resourcegroup="rg-zstream-upgrade-4-21-ssph6j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-fdsksh, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: failed waiting for cluster=\"cluster-zstream-4-21-fdsksh\" in resourcegroup=\"rg-zstream-upgrade-4-21-ssph6j\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012d10b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j",
            },
            <*fmt.wrapErrors | 0xc0009a8720>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-21-fdsksh\" in resourcegroup=\"rg-zstream-upgrade-4-21-ssph6j\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0012d10b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-fdsksh in resource group rg-zstream-upgrade-4-21-ssph6j",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Deserializaion Error: no output from command
00:23:16 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775525311 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
    STEP: creating a resource group @ 04/07/26 01:28:31.216
  "ts"="2026-04-07 01:28:31.216502" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-j4gv7q"
    STEP: creating cluster parameters @ 04/07/26 01:28:36.315
    STEP: creating customer resources @ 04/07/26 01:28:36.315
    STEP: creating the HCP cluster @ 04/07/26 01:29:58.448
  "ts"="2026-04-07 01:29:58.449091" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-j4gv7q"
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91 @ 04/07/26 02:14:58.449
  "ts"="2026-04-07 02:14:58.449977" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 02:14:58.450050" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-7c8wc2" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:14:58.733016" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-7c8wc2" "resourceGroup"=""
  "ts"="2026-04-07 02:15:02.047844" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 02:15:02.048091" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 02:15:02.048186" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 02:15:02.049536" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-16"
  "ts"="2026-04-07 02:15:02.049561" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:15:02.217441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-16" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 02:15:02.356982" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-16" "deployment"="msi-deployment-16"
  "ts"="2026-04-07 02:15:02.360274" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-j4gv7q" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
  "ts"="2026-04-07 02:15:02.682590" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-j4gv7q" "deployment"="customer-infra-cluster-np-labels-taints-7c8wc2"
  "ts"="2026-04-07 02:15:03.008994" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 02:15:03.009116" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 02:15:03.009272" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:24:38.523395" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:24:38.523422" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-labels-taints-j4gv7q"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775525311 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
    STEP: creating a resource group @ 04/07/26 01:28:31.216
  "ts"="2026-04-07 01:28:31.216502" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-j4gv7q"
    STEP: creating cluster parameters @ 04/07/26 01:28:36.315
    STEP: creating customer resources @ 04/07/26 01:28:36.315
    STEP: creating the HCP cluster @ 04/07/26 01:29:58.448
  "ts"="2026-04-07 01:29:58.449091" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-j4gv7q"
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91 @ 04/07/26 02:14:58.449
  "ts"="2026-04-07 02:14:58.449977" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 02:14:58.450050" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-7c8wc2" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:14:58.733016" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-7c8wc2" "resourceGroup"=""
  "ts"="2026-04-07 02:15:02.047844" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 02:15:02.048091" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-04-07 02:15:02.048186" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 02:15:02.049536" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-16"
  "ts"="2026-04-07 02:15:02.049561" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:15:02.217441" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-16" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 02:15:02.356982" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-16" "deployment"="msi-deployment-16"
  "ts"="2026-04-07 02:15:02.360274" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-j4gv7q" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
  "ts"="2026-04-07 02:15:02.682590" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-j4gv7q" "deployment"="customer-infra-cluster-np-labels-taints-7c8wc2"
  "ts"="2026-04-07 02:15:03.008994" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 02:15:03.009116" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 02:15:03.009272" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:24:38.523395" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-labels-taints-j4gv7q"
  "ts"="2026-04-07 02:24:38.523422" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-labels-taints-j4gv7q"
Command Error: signal: killed
Deserializaion Error: no output from command
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc001604390>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-nszsx5, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-nszsx5" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-nszsx5, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-nszsx5\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009757d0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-nszsx5",
            },
            <*fmt.wrapError | 0xc000e161a0>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-nszsx5\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc001604390>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-nszsx5, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-nszsx5" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-nszsx5, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-nszsx5\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009757d0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-nszsx5",
            },
            <*fmt.wrapError | 0xc000e161a0>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-nszsx5\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
00:23:16 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc0008d4210>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jwszks" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jwszks\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e02710>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks",
            },
            <*fmt.wrapErrors | 0xc0008d41e0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jwszks\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e02710>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc0008d4210>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jwszks" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jwszks\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e02710>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks",
            },
            <*fmt.wrapErrors | 0xc0008d41e0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jwszks\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e02710>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-jwszks",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Interrupted by User
00:23:16 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:184]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:184]: Interrupted by User
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e, unknown failed n/a 11
e2e (10)
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:13:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
    <*fmt.wrapErrors | 0xc00181f6e0>: 
    failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-jb4xcc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-jb4xcc, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jb4xcc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-jb4xcc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008dc420>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-jb4xcc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
    <*fmt.wrapErrors | 0xc00181f6e0>: 
    failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-jb4xcc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-jb4xcc, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jb4xcc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-jb4xcc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008dc420>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-jb4xcc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:13:02 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc001558fc0>: 
    failed waiting for hcpCluster="cluster-zstream-4-19-5cmtfg" in resourcegroup="rg-zstream-upgrade-4-19-tsqtqw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-5cmtfg in resource group rg-zstream-upgrade-4-19-tsqtqw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-5cmtfg\" in resourcegroup=\"rg-zstream-upgrade-4-19-tsqtqw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-5cmtfg in resource group rg-zstream-upgrade-4-19-tsqtqw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007fdb60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-5cmtfg in resource group rg-zstream-upgrade-4-19-tsqtqw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc001558fc0>: 
    failed waiting for hcpCluster="cluster-zstream-4-19-5cmtfg" in resourcegroup="rg-zstream-upgrade-4-19-tsqtqw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-5cmtfg in resource group rg-zstream-upgrade-4-19-tsqtqw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-5cmtfg\" in resourcegroup=\"rg-zstream-upgrade-4-19-tsqtqw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-5cmtfg in resource group rg-zstream-upgrade-4-19-tsqtqw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007fdb60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-19-5cmtfg in resource group rg-zstream-upgrade-4-19-tsqtqw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:13:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error:
    <*fmt.wrapErrors | 0xc0016a1620>: 
    failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-m987vc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-m987vc, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-m987vc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-m987vc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000765c80>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-m987vc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:155]: Unexpected error:
    <*fmt.wrapErrors | 0xc0016a1620>: 
    failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-m987vc" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-m987vc, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-m987vc\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-m987vc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000765c80>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-m987vc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Deserializaion Error: no output from command
00:13:02 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775524781 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 01:19:48.537
  "ts"="2026-04-07 01:19:48.537377" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-hw8d7b-thhkdf"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:19:52.971
    STEP: creating customer resources @ 04/07/26 01:19:53.063
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:21:09.322
  "ts"="2026-04-07 01:21:09.322669" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-hw8d7b" "resourceGroup"="rg-np-version-upgrade-hw8d7b-thhkdf"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 01:33:26.698
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775524781 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 01:19:48.537
  "ts"="2026-04-07 01:19:48.537377" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-hw8d7b-thhkdf"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:19:52.971
    STEP: creating customer resources @ 04/07/26 01:19:53.063
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:21:09.322
  "ts"="2026-04-07 01:21:09.322669" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-hw8d7b" "resourceGroup"="rg-np-version-upgrade-hw8d7b-thhkdf"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 01:33:26.698
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:13:02 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015307b0>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-grxshz" in resourcegroup="admin-credential-lifecycle-test-vdfx67" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-grxshz in resource group admin-credential-lifecycle-test-vdfx67, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-grxshz\" in resourcegroup=\"admin-credential-lifecycle-test-vdfx67\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-grxshz in resource group admin-credential-lifecycle-test-vdfx67, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a8c4b0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-grxshz in resource group admin-credential-lifecycle-test-vdfx67",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015307b0>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-grxshz" in resourcegroup="admin-credential-lifecycle-test-vdfx67" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-grxshz in resource group admin-credential-lifecycle-test-vdfx67, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-grxshz\" in resourcegroup=\"admin-credential-lifecycle-test-vdfx67\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-grxshz in resource group admin-credential-lifecycle-test-vdfx67, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a8c4b0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-grxshz in resource group admin-credential-lifecycle-test-vdfx67",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Interrupted by User
00:13:02 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:13:02 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc001365830>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-p9jg98" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-p9jg98, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-p9jg98\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-p9jg98, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000aeda50>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-p9jg98",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc001365830>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-p9jg98" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-p9jg98, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-p9jg98\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-p9jg98, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000aeda50>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-p9jg98",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:13:02 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
    <*fmt.wrapErrors | 0xc000dcc390>: 
    failed waiting for hcpCluster="cluster-stable-4-20-sv4k2p" in resourcegroup="rg-stable-4-20-sjfcpf" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-sv4k2p in resource group rg-stable-4-20-sjfcpf, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-sv4k2p\" in resourcegroup=\"rg-stable-4-20-sjfcpf\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-sv4k2p in resource group rg-stable-4-20-sjfcpf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a3e850>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-sv4k2p in resource group rg-stable-4-20-sjfcpf",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
    <*fmt.wrapErrors | 0xc000dcc390>: 
    failed waiting for hcpCluster="cluster-stable-4-20-sv4k2p" in resourcegroup="rg-stable-4-20-sjfcpf" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-sv4k2p in resource group rg-stable-4-20-sjfcpf, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-sv4k2p\" in resourcegroup=\"rg-stable-4-20-sjfcpf\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-sv4k2p in resource group rg-stable-4-20-sjfcpf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a3e850>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-sv4k2p in resource group rg-stable-4-20-sjfcpf",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Interrupted by User
00:13:02 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
00:13:02 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013c2210>: 
    failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-fqshxl" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-fqshxl, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-fqshxl\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-fqshxl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a96ca0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-fqshxl",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0013c2210>: 
    failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-fqshxl" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-fqshxl, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-fqshxl\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-fqshxl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a96ca0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-name-cluster in resource group patch-name-fqshxl",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
unknown (1)
Process did not finish before 2h0m0s timeout
00:13:02 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit
Show raw failure
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-07T02:16:09Z"}
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e, unknown failed n/a 2
e2e (1)
Deserializaion Error: no output from command
00:12:16 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775524025 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 01:07:05.535
  "ts"="2026-04-07 01:07:05.535184" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:07:08.385
    STEP: creating customer resources @ 04/07/26 01:07:08.466
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:08:01.242
  "ts"="2026-04-07 01:08:01.242928" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gg5w6p" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 01:21:28.227
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154 @ 04/07/26 02:06:28.229
  "ts"="2026-04-07 02:06:28.229894" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 02:06:28.229987" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-gg5w6p-86c4xv" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
  "ts"="2026-04-07 02:06:28.331005" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-gg5w6p-86c4xv" "resourceGroup"=""
  "ts"="2026-04-07 02:06:28.637569" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 02:06:28.637687" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 02:06:28.637757" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 02:06:28.639289" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-103"
  "ts"="2026-04-07 02:06:28.639301" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
  "ts"="2026-04-07 02:06:28.711446" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz" "deployment"="pooledNonMsiScopedAssignments-6bqq3eelft2oc"
  "ts"="2026-04-07 02:06:28.790571" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-103" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 02:06:28.870725" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz" "deployment"="customer-infra-np-version-upgrade-cluster-gg5w6p-86c4xv"
  "ts"="2026-04-07 02:06:28.939396" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-103" "deployment"="msi-deployment-103"
  "ts"="2026-04-07 02:06:29.124477" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 02:06:29.124591" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 02:06:29.124716" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775524025 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:234
    STEP: creating resource group @ 04/07/26 01:07:05.535
  "ts"="2026-04-07 01:07:05.535184" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
    STEP: creating cluster parameters at control plane version @ 04/07/26 01:07:08.385
    STEP: creating customer resources @ 04/07/26 01:07:08.466
    STEP: creating the HCP cluster with version 4.21.7 @ 04/07/26 01:08:01.242
  "ts"="2026-04-07 01:08:01.242928" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-gg5w6p" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
    STEP: creating nodepool with version 4.21.5 (behind control plane; upgrade path validated via Cincinnati) @ 04/07/26 01:21:28.227
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154 @ 04/07/26 02:06:28.229
  "ts"="2026-04-07 02:06:28.229894" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-07 02:06:28.229987" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-gg5w6p-86c4xv" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
  "ts"="2026-04-07 02:06:28.331005" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-gg5w6p-86c4xv" "resourceGroup"=""
  "ts"="2026-04-07 02:06:28.637569" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 02:06:28.637687" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-99031ad0c929f4b469b552900cdb42a4d2445ad8090c3f5e30a3cba67f4c8788.yaml.gz"
  "ts"="2026-04-07 02:06:28.637757" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-07 02:06:28.639289" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-103"
  "ts"="2026-04-07 02:06:28.639301" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
  "ts"="2026-04-07 02:06:28.711446" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz" "deployment"="pooledNonMsiScopedAssignments-6bqq3eelft2oc"
  "ts"="2026-04-07 02:06:28.790571" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-103" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-07 02:06:28.870725" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz" "deployment"="customer-infra-np-version-upgrade-cluster-gg5w6p-86c4xv"
  "ts"="2026-04-07 02:06:28.939396" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-103" "deployment"="msi-deployment-103"
  "ts"="2026-04-07 02:06:29.124477" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-07 02:06:29.124591" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-07 02:06:29.124716" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-gg5w6p-r2dqkz"
Command Error: signal: killed
Deserializaion Error: no output from command
unknown (1)
Process did not finish before 2h0m0s timeout
00:12:16 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit
Show raw failure
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-07T02:14:40Z"}