CIHealth
<
Single Day: Mar 1▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 13
e2e (13)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014888a0>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-2zrggd, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2zrggd" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-2zrggd, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2zrggd\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008f84a0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-2zrggd",
            },
            <*fmt.wrapError | 0xc000e925e0>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2zrggd\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014888a0>: 
    failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-2zrggd, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2zrggd" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-2zrggd, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2zrggd\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008f84a0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-2zrggd",
            },
            <*fmt.wrapError | 0xc000e925e0>{
                msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2zrggd\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:190]: Unexpected error:
    <*fmt.wrapError | 0xc0008101a0>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-w8wfrk" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-19-7\" for cluster \"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-w8wfrk\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:190]: Unexpected error:
    <*fmt.wrapError | 0xc0008101a0>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-w8wfrk" to finish creating: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-ver-4-19-7\" for cluster \"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-w8wfrk\" to finish creating: context deadline exceeded",
        err: <context.deadlineExceededError>{},
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011cc060>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-b4m6br, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-b4m6br" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-b4m6br, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b4m6br\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000b8c910>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-b4m6br",
            },
            <*fmt.wrapError | 0xc00080a6e0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b4m6br\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error:
    <*fmt.wrapErrors | 0xc0011cc060>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-b4m6br, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-b4m6br" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-b4m6br, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b4m6br\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000b8c910>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-b4m6br",
            },
            <*fmt.wrapError | 0xc00080a6e0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-b4m6br\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
Deserializaion Error: no output from command
02:00:52 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel
Show raw failure
odepool with autoscaling enabled (Spec Runtime: 1h29m59.958s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.958s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
          At [By Step] creating the autoscaling nodepool (Step Runtime: 19m22.214s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:95

          Spec Goroutine
          goroutine 103 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x820bb58, 0xc0008fc370}, 0x77f1f7b?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x824f7a0, {0x820bb58, 0xc0008fc370}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateNodePoolAndWait({0x820bbc8, 0xc000578930}, 0xc000095788, {0xc000f0ee10, 0x15}, {0x78183e3, 0x14}, {0x77f1f57, 0xc}, {0xc001523630, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:757
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateNodePoolFromParam(0xc0007b37a0, {0x7f78b44bda60, 0xc000a0b200}, {0xc000f0ee10, 0x15}, {0x78183e3, 0x14}, {{0x77d5d51, 0x6}, {0x78183e3, ...}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:384
          > github.com/Azure/ARO-HCP/test/e2e.init.func23.2({0x7f78b44bda60, 0xc000a0b200})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:96
                | 
                | By("creating the autoscaling nodepool")
                > err = tc.CreateNodePoolFromParam(ctx,
                | 	*resourceGroup.Name,
                | 	customerClusterName,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82213c0?, 0xc000a0b200?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102 @ 03/01/26 03:41:54.92
  "ts"="2026-03-01 03:41:54.920840" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-01 03:41:54.920909" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-q7mds5" "resourceGroup"="np-autoscaling-ntl9rp"
  "ts"="2026-03-01 03:41:55.097361" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-q7mds5" "resourceGroup"=""
  "ts"="2026-03-01 03:41:56.239629" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-01 03:41:56.239729" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-01 03:41:56.239818" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-01 03:41:56.240822" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-116"
  "ts"="2026-03-01 03:41:56.240836" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-ntl9rp"
  "ts"="2026-03-01 03:41:56.385178" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-116" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-01 03:41:56.408437" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-ntl9rp" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
  "ts"="2026-03-01 03:41:56.534892" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-116" "deployment"="msi-deployment-116"
  "ts"="2026-03-01 03:41:56.745301" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-ntl9rp" "deployment"="customer-infra-np-autoscale-cluster-q7mds5"
  "ts"="2026-03-01 03:41:56.911097" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-01 03:41:56.911200" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-01 03:41:56.911310" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-ntl9rp"
Command Error: signal: killed
Deserializaion Error: no output from command
odepool with autoscaling enabled (Spec Runtime: 1h29m59.958s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.958s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
          At [By Step] creating the autoscaling nodepool (Step Runtime: 19m22.214s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:95

          Spec Goroutine
          goroutine 103 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x820bb58, 0xc0008fc370}, 0x77f1f7b?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x824f7a0, {0x820bb58, 0xc0008fc370}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateNodePoolAndWait({0x820bbc8, 0xc000578930}, 0xc000095788, {0xc000f0ee10, 0x15}, {0x78183e3, 0x14}, {0x77f1f57, 0xc}, {0xc001523630, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:757
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateNodePoolFromParam(0xc0007b37a0, {0x7f78b44bda60, 0xc000a0b200}, {0xc000f0ee10, 0x15}, {0x78183e3, 0x14}, {{0x77d5d51, 0x6}, {0x78183e3, ...}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:384
          > github.com/Azure/ARO-HCP/test/e2e.init.func23.2({0x7f78b44bda60, 0xc000a0b200})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:96
                | 
                | By("creating the autoscaling nodepool")
                > err = tc.CreateNodePoolFromParam(ctx,
                | 	*resourceGroup.Name,
                | 	customerClusterName,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82213c0?, 0xc000a0b200?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102 @ 03/01/26 03:41:54.92
  "ts"="2026-03-01 03:41:54.920840" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-01 03:41:54.920909" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-q7mds5" "resourceGroup"="np-autoscaling-ntl9rp"
  "ts"="2026-03-01 03:41:55.097361" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-q7mds5" "resourceGroup"=""
  "ts"="2026-03-01 03:41:56.239629" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-01 03:41:56.239729" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-01 03:41:56.239818" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-01 03:41:56.240822" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-116"
  "ts"="2026-03-01 03:41:56.240836" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-ntl9rp"
  "ts"="2026-03-01 03:41:56.385178" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-116" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-01 03:41:56.408437" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-ntl9rp" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
  "ts"="2026-03-01 03:41:56.534892" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-116" "deployment"="msi-deployment-116"
  "ts"="2026-03-01 03:41:56.745301" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-ntl9rp" "deployment"="customer-infra-np-autoscale-cluster-q7mds5"
  "ts"="2026-03-01 03:41:56.911097" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-01 03:41:56.911200" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-01 03:41:56.911310" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-ntl9rp"
Command Error: signal: killed
Deserializaion Error: no output from command
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a684b0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-bqd84c, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-bqd84c" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-bqd84c, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-bqd84c\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00079fb10>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-bqd84c",
            },
            <*fmt.wrapError | 0xc0009921e0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-bqd84c\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a684b0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-bqd84c, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-bqd84c" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-bqd84c, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-bqd84c\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00079fb10>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-bqd84c",
            },
            <*fmt.wrapError | 0xc0009921e0>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-bqd84c\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:52 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d1f650>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-88jst9" in resourcegroup="admin-credential-lifecycle-test-sbtn87" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-88jst9 in resource group admin-credential-lifecycle-test-sbtn87, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-88jst9\" in resourcegroup=\"admin-credential-lifecycle-test-sbtn87\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-88jst9 in resource group admin-credential-lifecycle-test-sbtn87, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000d6b6f0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-88jst9 in resource group admin-credential-lifecycle-test-sbtn87",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d1f650>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-88jst9" in resourcegroup="admin-credential-lifecycle-test-sbtn87" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-88jst9 in resource group admin-credential-lifecycle-test-sbtn87, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-88jst9\" in resourcegroup=\"admin-credential-lifecycle-test-sbtn87\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-88jst9 in resource group admin-credential-lifecycle-test-sbtn87, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000d6b6f0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-88jst9 in resource group admin-credential-lifecycle-test-sbtn87",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc000efc060>: 
    failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xfxfjs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xfxfjs, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-xfxfjs\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xfxfjs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000b21af0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xfxfjs",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc000efc060>: 
    failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xfxfjs" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xfxfjs, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-xfxfjs\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xfxfjs, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000b21af0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xfxfjs",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc001106de0>: 
    failed waiting for external auth "ea-list" in resourcegroup="ea-list-6jmnbl" for cluster="ea-listr2rlb7" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ea-list in cluster ea-listr2rlb7 in resource group ea-list-6jmnbl, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"ea-list\" in resourcegroup=\"ea-list-6jmnbl\" for cluster=\"ea-listr2rlb7\" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ea-list in cluster ea-listr2rlb7 in resource group ea-list-6jmnbl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008d0390>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ea-list in cluster ea-listr2rlb7 in resource group ea-list-6jmnbl",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
    <*fmt.wrapErrors | 0xc001106de0>: 
    failed waiting for external auth "ea-list" in resourcegroup="ea-list-6jmnbl" for cluster="ea-listr2rlb7" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ea-list in cluster ea-listr2rlb7 in resource group ea-list-6jmnbl, error: context deadline exceeded
    {
        msg: "failed waiting for external auth \"ea-list\" in resourcegroup=\"ea-list-6jmnbl\" for cluster=\"ea-listr2rlb7\" to finish, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ea-list in cluster ea-listr2rlb7 in resource group ea-list-6jmnbl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008d0390>{
                s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ea-list in cluster ea-listr2rlb7 in resource group ea-list-6jmnbl",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Deserializaion Error: no output from command
02:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel
Show raw failure
custom autoscaling (Spec Runtime: 1h29m59.892s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.892s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:36
          At [By Step] creating the node pool (Step Runtime: 43m59.075s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:112

          Spec Goroutine
          goroutine 96 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x820bb58, 0xc000b26190}, 0x77f1f7b?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x824f7a0, {0x820bb58, 0xc000b26190}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateNodePoolAndWait({0x820bbc8, 0xc00072a1c0}, 0xc000095748, {0xc00127de00, 0x1a}, {0x78273c8, 0x17}, {0x77d2688, 0x4}, {0xc00157e570, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:757
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateNodePoolFromParam(0xc0007867e0, {0x7ffbb42ca660, 0xc000975b00}, {0xc00127de00, 0x1a}, {0x78273c8, 0x17}, {{0x77d5d51, 0x6}, {0x78273c8, ...}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:384
          > github.com/Azure/ARO-HCP/test/e2e.init.func6.2({0x7ffbb42ca660, 0xc000975b00})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:118
                | nodePoolParams.Replicas = int32(2)
                | 
                > err = tc.CreateNodePoolFromParam(ctx,
                | 	*resourceGroup.Name,
                | 	customerClusterName,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82213c0?, 0xc000975b00?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124 @ 03/01/26 03:41:53.656
  "ts"="2026-03-01 03:41:53.656795" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-01 03:41:53.656865" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-autoscaling-hcp-cluster-pfg4sv" "resourceGroup"="autoscaling-cluster-6mbww8"
  "ts"="2026-03-01 03:41:53.882690" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-autoscaling-hcp-cluster-pfg4sv" "resourceGroup"=""
  "ts"="2026-03-01 03:41:55.474450" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz"
  "ts"="2026-03-01 03:41:55.474569" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz"
  "ts"="2026-03-01 03:41:55.474638" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-01 03:41:55.475586" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-148"
  "ts"="2026-03-01 03:41:55.475602" "level"=0 "msg"="collecting deployments" "resourceGroup"="autoscaling-cluster-6mbww8"
  "ts"="2026-03-01 03:41:55.655617" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-6mbww8" "deployment"="pooledNonMsiScopedAssignments-e4ixwyxsfjnje"
  "ts"="2026-03-01 03:41:55.814591" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-6mbww8" "deployment"="customer-infra-autoscaling-hcp-cluster-pfg4sv"
  "ts"="2026-03-01 03:41:55.974500" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-01 03:41:56.147010" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="msi-deployment-148"
  "ts"="2026-03-01 03:41:56.300640" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-01 03:41:56.300813" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-01 03:41:56.300956" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="autoscaling-cluster-6mbww8"
Command Error: signal: killed
Deserializaion Error: no output from command
custom autoscaling (Spec Runtime: 1h29m59.892s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.892s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:36
          At [By Step] creating the node pool (Step Runtime: 43m59.075s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:112

          Spec Goroutine
          goroutine 96 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x820bb58, 0xc000b26190}, 0x77f1f7b?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x824f7a0, {0x820bb58, 0xc000b26190}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateNodePoolAndWait({0x820bbc8, 0xc00072a1c0}, 0xc000095748, {0xc00127de00, 0x1a}, {0x78273c8, 0x17}, {0x77d2688, 0x4}, {0xc00157e570, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:757
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateNodePoolFromParam(0xc0007867e0, {0x7ffbb42ca660, 0xc000975b00}, {0xc00127de00, 0x1a}, {0x78273c8, 0x17}, {{0x77d5d51, 0x6}, {0x78273c8, ...}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:384
          > github.com/Azure/ARO-HCP/test/e2e.init.func6.2({0x7ffbb42ca660, 0xc000975b00})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:118
                | nodePoolParams.Replicas = int32(2)
                | 
                > err = tc.CreateNodePoolFromParam(ctx,
                | 	*resourceGroup.Name,
                | 	customerClusterName,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82213c0?, 0xc000975b00?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:124 @ 03/01/26 03:41:53.656
  "ts"="2026-03-01 03:41:53.656795" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-01 03:41:53.656865" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-autoscaling-hcp-cluster-pfg4sv" "resourceGroup"="autoscaling-cluster-6mbww8"
  "ts"="2026-03-01 03:41:53.882690" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-autoscaling-hcp-cluster-pfg4sv" "resourceGroup"=""
  "ts"="2026-03-01 03:41:55.474450" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz"
  "ts"="2026-03-01 03:41:55.474569" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-31a71a35884d87a4a1de36eceeb324011e1c775154472da33ffc40cde64c08cb.yaml.gz"
  "ts"="2026-03-01 03:41:55.474638" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-01 03:41:55.475586" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-148"
  "ts"="2026-03-01 03:41:55.475602" "level"=0 "msg"="collecting deployments" "resourceGroup"="autoscaling-cluster-6mbww8"
  "ts"="2026-03-01 03:41:55.655617" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-6mbww8" "deployment"="pooledNonMsiScopedAssignments-e4ixwyxsfjnje"
  "ts"="2026-03-01 03:41:55.814591" "level"=0 "msg"="collecting operations" "resourceGroup"="autoscaling-cluster-6mbww8" "deployment"="customer-infra-autoscaling-hcp-cluster-pfg4sv"
  "ts"="2026-03-01 03:41:55.974500" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-01 03:41:56.147010" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-148" "deployment"="msi-deployment-148"
  "ts"="2026-03-01 03:41:56.300640" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-01 03:41:56.300813" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-01 03:41:56.300956" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="autoscaling-cluster-6mbww8"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:52 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e5a810>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wwvzjd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-wwvzjd, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wwvzjd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-wwvzjd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00072e2f0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-wwvzjd",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e5a810>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wwvzjd" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-wwvzjd, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wwvzjd\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-wwvzjd, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00072e2f0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-wwvzjd",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:120]: Unexpected error:
    <*fmt.wrapErrors | 0xc000cd2f00>: 
    failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-dsqvqh, error: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-dsqvqh" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-dsqvqh, error: failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-dsqvqh\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001423500>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-dsqvqh",
            },
            <*fmt.wrapError | 0xc000a015a0>{
                msg: "failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-dsqvqh\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:120]: Unexpected error:
    <*fmt.wrapErrors | 0xc000cd2f00>: 
    failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-dsqvqh, error: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-dsqvqh" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-dsqvqh, error: failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-dsqvqh\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001423500>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-dsqvqh",
            },
            <*fmt.wrapError | 0xc000a015a0>{
                msg: "failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-dsqvqh\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc000990840>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-wvfnxv, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wvfnxv" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-wvfnxv, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wvfnxv\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00136ba10>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-wvfnxv",
            },
            <*fmt.wrapError | 0xc000a76380>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wvfnxv\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error:
    <*fmt.wrapErrors | 0xc000990840>: 
    failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-wvfnxv, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wvfnxv" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-wvfnxv, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wvfnxv\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00136ba10>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-wvfnxv",
            },
            <*fmt.wrapError | 0xc000a76380>{
                msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wvfnxv\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:52 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:114]: Unexpected error:
    <*fmt.wrapErrors | 0xc000dd08a0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-5694vf, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-5694vf" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-5694vf, error: failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-5694vf\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009e2330>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-5694vf",
            },
            <*fmt.wrapError | 0xc000e04740>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-5694vf\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:114]: Unexpected error:
    <*fmt.wrapErrors | 0xc000dd08a0>: 
    failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-5694vf, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-5694vf" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-5694vf, error: failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-5694vf\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009e2330>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-5694vf",
            },
            <*fmt.wrapError | 0xc000e04740>{
                msg: "failed waiting for nodepool=\"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-5694vf\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred