CIHealth Failure Patterns

Environment: PROD

Failure patterns: 14 · Failures matched: 233 · Total runs: 41 · Runs affected: 30

Failure pattern

Failed at

Trend

Also in

Deserializaion Error: no output from command

full failure pattern:

Deserializaion Error: no output from command

Signal:

e2e

none

Full failure examples (3)

API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
  "ts"="2026-04-10 12:03:04.020611" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest|pid:880"
  "ts"="2026-04-10 12:03:04.028462" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating resource group @ 04/10/26 12:03:04.028
  "ts"="2026-04-10 12:03:04.028545" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
    STEP: creating cluster parameters at control plane version @ 04/10/26 12:03:08.089
W0410 12:03:08.168105     880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
    STEP: creating customer resources @ 04/10/26 12:03:08.168
    STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 12:04:03.752
  "ts"="2026-04-10 12:04:03.752222" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-wdq4gs" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
    STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:15:44.21
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:06.745
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:05.032
W0410 12:29:05.158022     880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
    STEP: capturing node release images before upgrade @ 04/10/26 12:29:05.158
    STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 12:29:05.194
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:48.069
    STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:49.726
  "ts"="2026-04-10 12:51:50.360556" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-10 12:51:50.360655" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 12:51:50.487211" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"=""
  "ts"="2026-04-10 12:51:51.762518" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
  "ts"="2026-04-10 12:51:51.762617" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
  "ts"="2026-04-10 12:51:51.762676" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-10 12:51:51.764204" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-131"
  "ts"="2026-04-10 12:51:51.764225" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 12:51:51.859754" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="pooledNonMsiScopedAssignments-dpbn3ikyauobc"
  "ts"="2026-04-10 12:51:52.188516" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7"
  "ts"="2026-04-10 12:51:52.242487" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-10 12:51:52.779918" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="msi-deployment-131"
  "ts"="2026-04-10 12:51:52.917289" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-10 12:51:52.917406" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-10 12:51:52.917517" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 13:02:10.173344" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 13:02:10.173371" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
Command Error: signal: killed
Deserializaion Error: no output from command
API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
  "ts"="2026-04-10 12:03:04.020611" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.21.zLatest|pid:880"
  "ts"="2026-04-10 12:03:04.028462" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating resource group @ 04/10/26 12:03:04.028
  "ts"="2026-04-10 12:03:04.028545" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
    STEP: creating cluster parameters at control plane version @ 04/10/26 12:03:08.089
W0410 12:03:08.168105     880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
    STEP: creating customer resources @ 04/10/26 12:03:08.168
    STEP: creating the HCP cluster with version 4.21.7 @ 04/10/26 12:04:03.752
  "ts"="2026-04-10 12:04:03.752222" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-wdq4gs" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
    STEP: creating nodepool with version 4.20.17 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:15:44.21
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:06.745
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:05.032
W0410 12:29:05.158022     880 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
    STEP: capturing node release images before upgrade @ 04/10/26 12:29:05.158
    STEP: triggering nodepool upgrade to version 4.21.7 @ 04/10/26 12:29:05.194
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:48.069
    STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:49.726
  "ts"="2026-04-10 12:51:50.360556" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-10 12:51:50.360655" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 12:51:50.487211" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-wdq4gs-95qms7" "resourceGroup"=""
  "ts"="2026-04-10 12:51:51.762518" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
  "ts"="2026-04-10 12:51:51.762617" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
  "ts"="2026-04-10 12:51:51.762676" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-10 12:51:51.764204" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-131"
  "ts"="2026-04-10 12:51:51.764225" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 12:51:51.859754" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="pooledNonMsiScopedAssignments-dpbn3ikyauobc"
  "ts"="2026-04-10 12:51:52.188516" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89" "deployment"="customer-infra-np-version-upgrade-cluster-wdq4gs-95qms7"
  "ts"="2026-04-10 12:51:52.242487" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-10 12:51:52.779918" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-131" "deployment"="msi-deployment-131"
  "ts"="2026-04-10 12:51:52.917289" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-10 12:51:52.917406" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-10 12:51:52.917517" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 13:02:10.173344" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
  "ts"="2026-04-10 13:02:10.173371" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-wdq4gs-qwpw89"
Command Error: signal: killed
Deserializaion Error: no output from command

Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775823245 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38
  "ts"="2026-04-10 12:14:05.070742" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings|pid:919"
  "ts"="2026-04-10 12:14:05.079100" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 04/10/26 12:14:05.079
  "ts"="2026-04-10 12:14:05.079151" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-7pqpv2"
    STEP: creating cluster parameters @ 04/10/26 12:14:09.56
    STEP: creating customer resources @ 04/10/26 12:14:09.56
    STEP: creating the HCP cluster @ 04/10/26 12:14:54.459
  "ts"="2026-04-10 12:14:54.459322" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-7pqpv2"
    STEP: creating a storage account for shoebox logs @ 04/10/26 12:29:05.918
  "ts"="2026-04-10 12:29:25.513737" "level"=0 "msg"="storage account created" "name"="shoeboxfpqdwk" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7pqpv2/providers/Microsoft.Storage/storageAccounts/shoeboxfpqdwk"
    STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 12:29:25.513
  "ts"="2026-04-10 12:29:30.702291" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting"
    STEP: waiting for log containers to appear in the storage account @ 04/10/26 12:29:30.702
  "ts"="2026-04-10 12:55:30.837118" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxfpqdwk" "timestamp"="2026-04-10T12:55:30Z"
  "ts"="2026-04-10 12:55:30.837292" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-10 12:55:30.837387" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf" "resourceGroup"="shoebox-cluster-7pqpv2"
  "ts"="2026-04-10 12:55:30.988434" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-j9gxkf" "resourceGroup"=""
  "ts"="2026-04-10 12:55:32.339790" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz"
  "ts"="2026-04-10 12:55:32.339933" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz"
  "ts"="2026-04-10 12:55:32.340053" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-10 12:55:32.341960" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-95"
  "ts"="2026-04-10 12:55:32.342015" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-7pqpv2"
  "ts"="2026-04-10 12:55:32.680479" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq"
  "ts"="2026-04-10 12:55:32.749543" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf"
  "ts"="2026-04-10 12:55:32.952168" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-10 12:55:33.567562" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="msi-deployment-95"
  "ts"="2026-04-10 12:55:33.845873" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-10 12:55:33.846012" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-10 12:55:33.846126" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-7pqpv2"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1775823245 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, Stage-And-Prod-Only]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:38
  "ts"="2026-04-10 12:14:05.070742" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-be-able-to-forward-control-plane-logs-to-a-storage-account-via-shoebox-diagnostic-settings|pid:919"
  "ts"="2026-04-10 12:14:05.079100" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 04/10/26 12:14:05.079
  "ts"="2026-04-10 12:14:05.079151" "level"=0 "msg"="creating resource group" "resourceGroup"="shoebox-cluster-7pqpv2"
    STEP: creating cluster parameters @ 04/10/26 12:14:09.56
    STEP: creating customer resources @ 04/10/26 12:14:09.56
    STEP: creating the HCP cluster @ 04/10/26 12:14:54.459
  "ts"="2026-04-10 12:14:54.459322" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="shoebox-hcp-cluster" "resourceGroup"="shoebox-cluster-7pqpv2"
    STEP: creating a storage account for shoebox logs @ 04/10/26 12:29:05.918
  "ts"="2026-04-10 12:29:25.513737" "level"=0 "msg"="storage account created" "name"="shoeboxfpqdwk" "id"="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7pqpv2/providers/Microsoft.Storage/storageAccounts/shoeboxfpqdwk"
    STEP: enabling diagnostic settings on the HCP cluster @ 04/10/26 12:29:25.513
  "ts"="2026-04-10 12:29:30.702291" "level"=0 "msg"="diagnostic setting created" "name"="shoebox-diag-setting"
    STEP: waiting for log containers to appear in the storage account @ 04/10/26 12:29:30.702
  "ts"="2026-04-10 12:55:30.837118" "level"=0 "msg"="shoebox log containers found in storage account" "storageAccount"="shoeboxfpqdwk" "timestamp"="2026-04-10T12:55:30Z"
  "ts"="2026-04-10 12:55:30.837292" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-10 12:55:30.837387" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf" "resourceGroup"="shoebox-cluster-7pqpv2"
  "ts"="2026-04-10 12:55:30.988434" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-shoebox-hcp-cluster-j9gxkf" "resourceGroup"=""
  "ts"="2026-04-10 12:55:32.339790" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz"
  "ts"="2026-04-10 12:55:32.339933" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-28def4d2520d754dc684cda51e7fbbbda7eafe4c1b56787244f7f5c8f8c5f740.yaml.gz"
  "ts"="2026-04-10 12:55:32.340053" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-10 12:55:32.341960" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-95"
  "ts"="2026-04-10 12:55:32.342015" "level"=0 "msg"="collecting deployments" "resourceGroup"="shoebox-cluster-7pqpv2"
  "ts"="2026-04-10 12:55:32.680479" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="pooledNonMsiScopedAssignments-22eudp3sazqeq"
  "ts"="2026-04-10 12:55:32.749543" "level"=0 "msg"="collecting operations" "resourceGroup"="shoebox-cluster-7pqpv2" "deployment"="customer-infra-shoebox-hcp-cluster-j9gxkf"
  "ts"="2026-04-10 12:55:32.952168" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-10 12:55:33.567562" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-95" "deployment"="msi-deployment-95"
  "ts"="2026-04-10 12:55:33.845873" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-10 12:55:33.846012" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-10 12:55:33.846126" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="shoebox-cluster-7pqpv2"
Command Error: signal: killed
Deserializaion Error: no output from command

t/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226
W0410 12:04:14.665210     893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
W0410 12:04:14.748424     893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
  "ts"="2026-04-10 12:04:14.748740" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest|pid:893"
  "ts"="2026-04-10 12:04:14.933427" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating resource group @ 04/10/26 12:04:14.933
  "ts"="2026-04-10 12:04:14.933646" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
    STEP: creating cluster parameters at control plane version @ 04/10/26 12:04:19.473
    STEP: creating customer resources @ 04/10/26 12:04:19.579
    STEP: creating the HCP cluster with version 4.20.17 @ 04/10/26 12:05:04.201
  "ts"="2026-04-10 12:05:04.201160" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-jnm8s9" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
    STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:16:02.227
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:01.442
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:09.867
    STEP: capturing node release images before upgrade @ 04/10/26 12:29:10.041
    STEP: triggering nodepool upgrade to version 4.20.17 @ 04/10/26 12:29:10.08
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:52.504
    STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:54.086
  "ts"="2026-04-10 12:51:54.286433" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-10 12:51:54.286512" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 12:51:54.379693" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"=""
  "ts"="2026-04-10 12:51:55.257214" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-10 12:51:55.257316" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-10 12:51:55.257391" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-10 12:51:55.259038" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-129"
  "ts"="2026-04-10 12:51:55.259056" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 12:51:55.353380" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="pooledNonMsiScopedAssignments-jfl6npesxtst4"
  "ts"="2026-04-10 12:51:55.415329" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w"
  "ts"="2026-04-10 12:51:55.578837" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-10 12:51:55.709848" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="msi-deployment-129"
  "ts"="2026-04-10 12:51:55.860303" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-10 12:51:55.860430" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-10 12:51:55.860601" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 13:02:14.746040" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 13:02:14.746065" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
Command Error: signal: killed
Deserializaion Error: no output from command
t/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:226
W0410 12:04:14.665210     893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
W0410 12:04:14.748424     893 cincinnati.go:308] Conditional update to 4.21.8, risk "PrecisionTimeProtocolDPLLPins", has empty pruned matchingRules; dropping this target to avoid rejections when pushing to the Kubernetes API server. Pruning results: skipping unrecognized cluster condition type "PromQL"
  "ts"="2026-04-10 12:04:14.748740" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Customer-should-upgrade-a-nodepool-from-4.20.z-to-4.20.zLatest|pid:893"
  "ts"="2026-04-10 12:04:14.933427" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating resource group @ 04/10/26 12:04:14.933
  "ts"="2026-04-10 12:04:14.933646" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
    STEP: creating cluster parameters at control plane version @ 04/10/26 12:04:19.473
    STEP: creating customer resources @ 04/10/26 12:04:19.579
    STEP: creating the HCP cluster with version 4.20.17 @ 04/10/26 12:05:04.201
  "ts"="2026-04-10 12:05:04.201160" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-jnm8s9" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
    STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 04/10/26 12:16:02.227
    STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 04/10/26 12:26:01.442
    STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 04/10/26 12:29:09.867
    STEP: capturing node release images before upgrade @ 04/10/26 12:29:10.041
    STEP: triggering nodepool upgrade to version 4.20.17 @ 04/10/26 12:29:10.08
    STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 04/10/26 12:29:52.504
    STEP: verifying node pool GET still reflects the new version @ 04/10/26 12:51:54.086
  "ts"="2026-04-10 12:51:54.286433" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-04-10 12:51:54.286512" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 12:51:54.379693" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-jnm8s9-rpjd2w" "resourceGroup"=""
  "ts"="2026-04-10 12:51:55.257214" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-10 12:51:55.257316" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-06e5d303cf312917255f4ef7fc1bf912c96515354295cfec0e753053c32d9eb0.yaml.gz"
  "ts"="2026-04-10 12:51:55.257391" "level"=0 "msg"="collecting debug info"
  "ts"="2026-04-10 12:51:55.259038" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-129"
  "ts"="2026-04-10 12:51:55.259056" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 12:51:55.353380" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="pooledNonMsiScopedAssignments-jfl6npesxtst4"
  "ts"="2026-04-10 12:51:55.415329" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5" "deployment"="customer-infra-np-version-upgrade-cluster-jnm8s9-rpjd2w"
  "ts"="2026-04-10 12:51:55.578837" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-04-10 12:51:55.709848" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-129" "deployment"="msi-deployment-129"
  "ts"="2026-04-10 12:51:55.860303" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-04-10 12:51:55.860430" "level"=0 "msg"="deleting created resources"
  "ts"="2026-04-10 12:51:55.860601" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 13:02:14.746040" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
  "ts"="2026-04-10 13:02:14.746065" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-jnm8s9-tb54l5"
Command Error: signal: killed
Deserializaion Error: no output from command

Contributing tests (22)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	9
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	7
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to update node pool labels and taints	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest	5
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	1

Affected runs (15)

Date (UTC)	Associated PR	Prow job
2026-04-10 10:59	n/a	2042558518153711616
2026-04-10 10:57	n/a	2042557690818859008
2026-04-10 08:42	n/a	2042523562169864192
2026-04-10 08:37	n/a	2042522288930164736
2026-04-10 02:06	n/a	2042423946946547712
2026-04-09 23:42	n/a	2042387582653829120
2026-04-09 02:13	n/a	2042063253525237760
2026-04-09 02:00	n/a	2042060062305816576
2026-04-07 04:05	n/a	2041366836473237504
2026-04-07 04:05	n/a	2041366674841538560
2026-04-07 04:04	n/a	2041366521514561536
2026-04-07 00:23	n/a	2041310807207186432
2026-04-07 00:13	n/a	2041308229962567680
2026-04-07 00:12	n/a	2041308037523705856
2026-04-06 18:04	n/a	2041215587668987904

Interrupted by User

full failure pattern:

Interrupted by User

Signal:

e2e

none

Full failure examples (3)

fail [:0]: Interrupted by User
fail [:0]: Interrupted by User

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:44]: Interrupted by User

Contributing tests (13)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group	7
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.23	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to list HCP clusters without node pools at the resource group level	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1

Affected runs (15)

Date (UTC)	Associated PR	Prow job
2026-04-12 07:50	n/a	2043235187642339328
2026-04-09 10:43	n/a	2042191604369854464
2026-04-09 02:13	n/a	2042063253525237760
2026-04-09 02:00	n/a	2042060062305816576
2026-04-08 15:56	n/a	2041908135588794368
2026-04-08 13:08	n/a	2041865685600047104
2026-04-07 04:05	n/a	2041366836473237504
2026-04-07 04:04	n/a	2041366521514561536
2026-04-07 00:23	n/a	2041310807207186432
2026-04-07 00:13	n/a	2041308229962567680
2026-04-06 22:17	n/a	2041279040924422144
2026-04-06 21:58	n/a	2041274391769124864
2026-04-06 21:47	n/a	2041271574195081216
2026-04-06 18:04	n/a	2041215587668987904
2026-04-06 08:46	n/a	2041074948453175296

Process did not finish before 2h0m0s timeout

full failure pattern:

Process did not finish before 2h0m0s timeout

Signal:

unknown

none

Full failure examples (3)

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T13:03:33Z"}

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T13:01:55Z"}

{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-04-10T10:44:38Z"}

Contributing tests (1)

Failed At	Job	Test	Support
unknown	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Job run should complete before timeout	14

Affected runs (14)

Date (UTC)	Associated PR	Prow job
2026-04-10 10:59	n/a	2042558518153711616
2026-04-10 10:57	n/a	2042557690818859008
2026-04-10 08:42	n/a	2042523562169864192
2026-04-10 08:37	n/a	2042522288930164736
2026-04-10 02:06	n/a	2042423946946547712
2026-04-09 23:42	n/a	2042387582653829120
2026-04-09 02:13	n/a	2042063253525237760
2026-04-09 02:00	n/a	2042060062305816576
2026-04-07 04:05	n/a	2041366836473237504
2026-04-07 04:05	n/a	2041366674841538560
2026-04-07 04:04	n/a	2041366521514561536
2026-04-07 00:13	n/a	2041308229962567680
2026-04-07 00:12	n/a	2041308037523705856
2026-04-06 18:04	n/a	2041215587668987904

timeout during CreateHCPClusterAndWait; context deadline exceeded

full failure pattern:

timeout during CreateHCPClusterAndWait; context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e54480>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-9rklxc, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-9rklxc in resource group rg-cp-ystream-upgrade-4-21-rqz9bx, error: failed waiting for cluster="cp-ystream-upgrade-4-21-9rklxc" in resourcegroup="rg-cp-ystream-upgrade-4-21-rqz9bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-9rklxc in resource group rg-cp-ystream-upgrade-4-21-rqz9bx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e54480>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-9rklxc, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-9rklxc in resource group rg-cp-ystream-upgrade-4-21-rqz9bx, error: failed waiting for cluster="cp-ystream-upgrade-4-21-9rklxc" in resourcegroup="rg-cp-ystream-upgrade-4-21-rqz9bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cp-ystream-upgrade-4-21-9rklxc in resource group rg-cp-ystream-upgrade-4-21-rqz9bx, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00150c090>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-c9kcwx, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-c9kcwx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-c9kcwx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00150c090>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-c9kcwx, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-c9kcwx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-c9kcwx, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc001727110>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nftcd7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nftcd7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc001727110>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nftcd7" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nftcd7, error: context deadline exceeded
    ...
occurred

Contributing tests (27)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should respect cluster-wide node limits with nodepool autoscaling	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to forward control plane logs to a storage account via shoebox diagnostic settings	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to lifecycle and confirm external auth on a cluster	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to update node pool labels and taints	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to reuse subnets and NSGs between clusters	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest	1

Affected runs (10)

Date (UTC)	Associated PR	Prow job
2026-04-12 22:01	n/a	2043449495512944640
2026-04-12 07:50	n/a	2043235269712285696
2026-04-12 07:50	n/a	2043235187642339328
2026-04-10 20:29	n/a	2042701416304218112
2026-04-10 16:55	n/a	2042647665363128320
2026-04-10 16:55	n/a	2042647552989335552
2026-04-10 02:06	n/a	2042423946946547712
2026-04-09 23:42	n/a	2042387582653829120
2026-04-07 00:23	n/a	2041310807207186432
2026-04-06 21:58	n/a	2041274391769124864

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

full failure pattern:

timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error:
    <*fmt.wrapErrors | 0xc00109e3c0>: 
    failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-9r68hs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-9r68hs, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error:
    <*fmt.wrapErrors | 0xc00109e3c0>: 
    failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-9r68hs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-9r68hs, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc00158c090>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-l2hzs5" in resourcegroup="admin-credential-lifecycle-test-lsrgg4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-l2hzs5 in resource group admin-credential-lifecycle-test-lsrgg4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc00158c090>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-l2hzs5" in resourcegroup="admin-credential-lifecycle-test-lsrgg4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-l2hzs5 in resource group admin-credential-lifecycle-test-lsrgg4, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc00170fdd0>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qwvv9r" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-qwvv9r, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc00170fdd0>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qwvv9r" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-qwvv9r, error: context deadline exceeded
    ...
occurred

Contributing tests (17)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	6
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	4
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage ImageDigestMirrors	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should not be able to perform various invalid operations on cluster resources	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20	1

Affected runs (10)

Date (UTC)	Associated PR	Prow job
2026-04-12 07:50	n/a	2043235187642339328
2026-04-10 20:29	n/a	2042701416304218112
2026-04-10 19:22	n/a	2042684774031560704
2026-04-10 16:55	n/a	2042647665363128320
2026-04-10 10:57	n/a	2042557690818859008
2026-04-10 02:06	n/a	2042423946946547712
2026-04-09 02:13	n/a	2042063253525237760
2026-04-09 02:00	n/a	2042060062305816576
2026-04-07 00:23	n/a	2041310807207186432
2026-04-07 00:13	n/a	2041308229962567680

ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapError | 0xc000e9c280>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rtxghs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-rp69dr/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapError | 0xc000e9c280>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rtxghs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-rp69dr/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc001380380>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-pb6kcg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc001380380>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-pb6kcg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapError | 0xc000d2c120>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-xgtzgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapError | 0xc000d2c120>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-xgtzgf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (14)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled	3
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with back-level version 4.19	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to update node pool labels and taints	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request	1

Affected runs (9)

Date (UTC)	Associated PR	Prow job
2026-04-12 07:50	n/a	2043235269712285696
2026-04-12 07:50	n/a	2043235187642339328
2026-04-10 20:29	n/a	2042701416304218112
2026-04-10 16:55	n/a	2042647552989335552
2026-04-10 10:57	n/a	2042557690818859008
2026-04-10 02:06	n/a	2042423946946547712
2026-04-09 10:43	n/a	2042191604369854464
2026-04-07 04:04	n/a	2041366521514561536
2026-04-06 08:46	n/a	2041074948453175296

CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded

full failure pattern:

CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc00130e390>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-czfgf5, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-czfgf5" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc00130e390>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-czfgf5, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-czfgf5" to finish creating: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154]: Unexpected error:
    <*fmt.wrapErrors | 0xc000958690>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-pmp787" in resourcegroup="rg-np-version-upgrade-pmp787-kwgkwl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001248460>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl",
            },
            <*fmt.wrapError | 0xc001104340>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:154]: Unexpected error:
    <*fmt.wrapErrors | 0xc000958690>: 
    failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-pmp787" in resourcegroup="rg-np-version-upgrade-pmp787-kwgkwl" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-21, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl, error: failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001248460>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-21 in resource group rg-np-version-upgrade-pmp787-kwgkwl",
            },
            <*fmt.wrapError | 0xc001104340>{
                msg: "failed waiting for nodepool=\"npupgrade-4-21\" for cluster \"np-version-upgrade-cluster-pmp787\" in resourcegroup=\"rg-np-version-upgrade-pmp787-kwgkwl\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012d86f0>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-b8kv4h" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001247440>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h",
            },
            <*fmt.wrapError | 0xc0005187e0>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012d86f0>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-b8kv4h" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001247440>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-b8kv4h",
            },
            <*fmt.wrapError | 0xc0005187e0>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-b8kv4h\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred

Contributing tests (6)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and custom node pool osDisk size	2
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster and manage pull secrets	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create node pools with ARM64-based VMs	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should create an HCP cluster and validate TLS certificates	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.21.z to 4.21.zLatest	1

Affected runs (3)

Date (UTC)	Associated PR	Prow job
2026-04-12 07:50	n/a	2043235187642339328
2026-04-07 04:05	n/a	2041366674841538560
2026-04-07 00:23	n/a	2041310807207186432

tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>

full failure pattern:

tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>

Signal:

e2e

none

Full failure examples (3)

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:107]: Unexpected error:
    <*errors.joinError | 0xc000b2f950>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ea-cluster.50y5.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.ea-cluster.kv02.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.ea-cluster.50y5.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ea-cluster.50y5.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.ea-cluster.kv02.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.ea-cluster.50y5.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:107]: Unexpected error:
    <*errors.joinError | 0xc000b2f950>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ea-cluster.50y5.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.ea-cluster.kv02.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.ea-cluster.50y5.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ea-cluster.50y5.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.ea-cluster.kv02.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.ea-cluster.50y5.uksouth.aroapp-hcp.io
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:168]: Unexpected error:
    <*url.Error | 0xc001248ab0>: 
    Get "https://api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.q6i1z1a9v3j3w8r.qgw8.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:168]: Unexpected error:
    <*url.Error | 0xc001248ab0>: 
    Get "https://api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.q6i1z1a9v3j3w8r.qgw8.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.q6i1z1a9v3j3w8r.lr4g.uksouth.aroapp-hcp.io
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
    <*errors.joinError | 0xc0013be210>: 
    VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
    <*errors.joinError | 0xc0013be210>: 
    VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io
    VerifyBasicAccess failed: failed to list services: Get "https://api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.j5b3t8t5w8i3j5x.8h0z.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.j5b3t8t5w8i3j5x.83aa.uksouth.aroapp-hcp.io
    ...
occurred

Contributing tests (3)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create a cluster with an external auth config and get the external auth config	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with Image Registry not present	1
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest	1

Affected runs (2)

Date (UTC)	Associated PR	Prow job
2026-04-12 22:01	n/a	2043449495512944640
2026-04-09 02:13	n/a	2042063253525237760

Cluster should become ready within 45 minutes

full failure pattern:

Cluster should become ready within 45 minutes

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: ...
to be true

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-10 02:06	n/a	2042423946946547712

ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:85]: Unexpected error:
    <*fmt.wrapError | 0xc0001769a0>: 
    failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation "list-test-cluster-p4ct5p" in resourcegroup="cluster-listing-m5pqxl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000176980>{
            msg: "failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000c6f590>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Mon, 06 Apr 2026 18:28:48 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/f8dcff39-92dd-4b53-bfd5-a54e320e9c99",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260406T182849Z:c7e70c50-1876-4215-b20d-f5fe6a8b0f35",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 6C605D2072634CEB987C0EC2A7E6913D Ref B: CH1AA2020603011 Ref C: 2026-04-06T18:28:45Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "057ea031-c736-4cfd-82cb-048ffa7ce574",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "c7e70c50-18...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:85]: Unexpected error:
    <*fmt.wrapError | 0xc0001769a0>: 
    failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation "list-test-cluster-p4ct5p" in resourcegroup="cluster-listing-m5pqxl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster list-test-cluster-p4ct5p: failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000176980>{
            msg: "failed starting cluster creation \"list-test-cluster-p4ct5p\" in resourcegroup=\"cluster-listing-m5pqxl\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Cannot update resource while resource is \\\"accepted\\\"\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-m5pqxl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-p4ct5p\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000c6f590>{
                ErrorCode: "Conflict",
                StatusCode: 409,
                RawResponse: {
                    Status: "409 Conflict",
                    StatusCode: 409,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Mon, 06 Apr 2026 18:28:48 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/f8dcff39-92dd-4b53-bfd5-a54e320e9c99",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260406T182849Z:c7e70c50-1876-4215-b20d-f5fe6a8b0f35",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 6C605D2072634CEB987C0EC2A7E6913D Ref B: CH1AA2020603011 Ref C: 2026-04-06T18:28:45Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Error-Code": ["Conflict"],
                        "X-Ms-Request-Id": [
                            "057ea031-c736-4cfd-82cb-048ffa7ce574",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "c7e70c50-18...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to list HCP clusters without node pools at the resource group level	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-06 18:04	n/a	2041215587668987904

ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' has an external authentication pending deletion; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: InvalidRequestContent; detail message Cluster '<id>' has an external authentication pending deletion; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:294]: Unexpected error:
    <*fmt.wrapError | 0xc001508c80>: 
    failed creating external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-n5j64n" for cluster="cidr-connectivity-test": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n5j64n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/externalAuths/ext-auth-cidr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Cluster '2pj8n2ov52hfcbj1lfepdkk77i8o48uj' has an external authentication pending deletion"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:294]: Unexpected error:
    <*fmt.wrapError | 0xc001508c80>: 
    failed creating external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-n5j64n" for cluster="cidr-connectivity-test": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n5j64n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test/externalAuths/ext-auth-cidr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Cluster '2pj8n2ov52hfcbj1lfepdkk77i8o48uj' has an external authentication pending deletion"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Authorized CIDRs Connectivity should allow API access only from authorized VM IP	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-10 10:59	n/a	2042558518153711616

ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift

full failure pattern:

ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*exported.ResponseError | 0xc001404990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rclvn7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*exported.ResponseError | 0xc001404990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rclvn7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to create an HCP cluster with custom autoscaling	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-09 10:43	n/a	2042191604369854464

get deployment router-default -n openshift-ingress: Get "<url>

full failure pattern:

get deployment router-default -n openshift-ingress: Get "<url>

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:180]: Timed out after 600.000s.
router-default deployment logs should be fetchable
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d582a0>: 
    get deployment router-default -n openshift-ingress: Get "https://api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io:443/apis/apps/v1/namespaces/openshift-ingress/deployments/router-default": tls: failed to verify certificate: x509: certificate is valid for api.o6n1s5y5u2a8q7c.wjok.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:180]: Timed out after 600.000s.
router-default deployment logs should be fetchable
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d582a0>: 
    get deployment router-default -n openshift-ingress: Get "https://api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io:443/apis/apps/v1/namespaces/openshift-ingress/deployments/router-default": tls: failed to verify certificate: x509: certificate is valid for api.o6n1s5y5u2a8q7c.wjok.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.o6n1s5y5u2a8q7c.i7sr.uksouth.aroapp-hcp.io
    ...

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-09 02:00	n/a	2042060062305816576

pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range

full failure pattern:

pod "<pod>" is forbidden: error fetching namespace "<namespace>": unable to find annotation openshift.io/sa.scc.uid-range

Signal:

e2e

none

Full failure examples (1)

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc0012b7ae0>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:344]: Unexpected error:
    <*errors.StatusError | 0xc0012b7ae0>: 
    pods "wi-test-pod" is forbidden: error fetching namespace "e2e-oidc-wi": unable to find annotation openshift.io/sa.scc.uid-range
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "pods \"wi-test-pod\" is forbidden: error fetching namespace \"e2e-oidc-wi\": unable to find annotation openshift.io/sa.scc.uid-range",
            Reason: "Forbidden",
            Details: {Name: "wi-test-pod", Group: "", Kind: "pods", UID: "", Causes: nil, RetryAfterSeconds: 0},
            Code: 403,
        },
    }
occurred

Contributing tests (1)

Failed At	Job	Test	Support
e2e	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel	Customer should be able to use workload identity via the cluster OIDC issuer URL	1

Affected runs (1)

Date (UTC)	Associated PR	Prow job
2026-04-06 00:31	n/a	2040950439863652352