<
Single Day: Jul 6
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 5
e2e (5)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
23:00:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-rw5292" for cluster "cluster-nightly-4-20-rw5292"
Unexpected error:
    <*fmt.wrapErrors | 0xc00251f860>: 
    failed to create NodePool np-rw5292, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-rw5292 in resource group rg-nightly-4-20-g8lx9j, error: failed waiting for nodepool="np-rw5292" for cluster "cluster-nightly-4-20-rw5292" in resourcegroup="rg-nightly-4-20-g8lx9j" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-rw5292" for cluster "cluster-nightly-4-20-rw5292"
Unexpected error:
    <*fmt.wrapErrors | 0xc00251f860>: 
    failed to create NodePool np-rw5292, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-rw5292 in resource group rg-nightly-4-20-g8lx9j, error: failed waiting for nodepool="np-rw5292" for cluster "cluster-nightly-4-20-rw5292" in resourcegroup="rg-nightly-4-20-g8lx9j" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:00:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-5ghdp7/cluster-nightly-5-0-r2rdcg should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc000909e90>: 
    failed to create HCP cluster cluster-nightly-5-0-r2rdcg, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-r2rdcg in resource group rg-nightly-5-0-5ghdp7, error: failed waiting for cluster="cluster-nightly-5-0-r2rdcg" in resourcegroup="rg-nightly-5-0-5ghdp7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-r2rdcg in resource group rg-nightly-5-0-5ghdp7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-5ghdp7/cluster-nightly-5-0-r2rdcg should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc000909e90>: 
    failed to create HCP cluster cluster-nightly-5-0-r2rdcg, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-r2rdcg in resource group rg-nightly-5-0-5ghdp7, error: failed waiting for cluster="cluster-nightly-5-0-r2rdcg" in resourcegroup="rg-nightly-5-0-5ghdp7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-r2rdcg in resource group rg-nightly-5-0-5ghdp7, error: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
23:00:20 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0017a2468>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-r77729, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-r77729" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0017a2468>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-r77729, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-r77729" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:00:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-23-g897nm/cluster-nightly-4-23-gjksrw should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001076ae0>: 
    failed to create HCP cluster cluster-nightly-4-23-gjksrw, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-gjksrw in resource group rg-nightly-4-23-g897nm, error: failed waiting for cluster="cluster-nightly-4-23-gjksrw" in resourcegroup="rg-nightly-4-23-g897nm" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-gjksrw in resource group rg-nightly-4-23-g897nm, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-23-g897nm/cluster-nightly-4-23-gjksrw should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001076ae0>: 
    failed to create HCP cluster cluster-nightly-4-23-gjksrw, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-gjksrw in resource group rg-nightly-4-23-g897nm, error: failed waiting for cluster="cluster-nightly-4-23-gjksrw" in resourcegroup="rg-nightly-4-23-g897nm" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-gjksrw in resource group rg-nightly-4-23-g897nm, error: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
23:00:20 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc002a91530>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-lxfx2v, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-lxfx2v" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc002a91530>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-lxfx2v, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-lxfx2v" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 4
e2e (4)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:04:45 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-kk6pb6" for cluster "cluster-candidate-4-20-kk6pb6"
Unexpected error:
    <*fmt.wrapErrors | 0xc0024c3b60>: 
    failed to create NodePool np-kk6pb6, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-kk6pb6 in resource group rg-candidate-4-20-8gx7xt, error: failed waiting for nodepool="np-kk6pb6" for cluster "cluster-candidate-4-20-kk6pb6" in resourcegroup="rg-candidate-4-20-8gx7xt" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-kk6pb6" for cluster "cluster-candidate-4-20-kk6pb6"
Unexpected error:
    <*fmt.wrapErrors | 0xc0024c3b60>: 
    failed to create NodePool np-kk6pb6, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-kk6pb6 in resource group rg-candidate-4-20-8gx7xt, error: failed waiting for nodepool="np-kk6pb6" for cluster "cluster-candidate-4-20-kk6pb6" in resourcegroup="rg-candidate-4-20-8gx7xt" to finish creating: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:04:45 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0027db170>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-5qkz7d, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5qkz7d" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0027db170>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-5qkz7d, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5qkz7d" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
12:04:45 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-wg275n/cluster-candidate-5-0-tcf7s2 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00179bc50>: 
    failed to create HCP cluster cluster-candidate-5-0-tcf7s2, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-tcf7s2 in resource group rg-candidate-5-0-wg275n, error: failed waiting for cluster="cluster-candidate-5-0-tcf7s2" in resourcegroup="rg-candidate-5-0-wg275n" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-tcf7s2 in resource group rg-candidate-5-0-wg275n, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-wg275n/cluster-candidate-5-0-tcf7s2 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00179bc50>: 
    failed to create HCP cluster cluster-candidate-5-0-tcf7s2, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-tcf7s2 in resource group rg-candidate-5-0-wg275n, error: failed waiting for cluster="cluster-candidate-5-0-tcf7s2" in resourcegroup="rg-candidate-5-0-wg275n" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-tcf7s2 in resource group rg-candidate-5-0-wg275n, error: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:04:45 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002b2e498>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-vjgmkx, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-vjgmkx" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002b2e498>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-vjgmkx, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-vjgmkx" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 23
e2e (23)
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 503 Service Unavailable: <html>
07:08:34 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 503 Service Unavailable: <html>
07:08:34 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
07:08:34 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:677]: failed to get latest version for minor 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0010be160>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:677]: failed to get latest version for minor 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0010be160>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
    ...
occurred
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
07:08:34 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:536]: failed to get versions for minor 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000e1c460>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:536]: failed to get versions for minor 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000e1c460>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
07:08:34 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001122408>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-k5jvws, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-k5jvws" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001122408>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-k5jvws, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-k5jvws" to finish creating: context deadline exceeded
    ...
occurred
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
query graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:257]: failed to get version pair without upgrade edge for minor 4.20
Unexpected error:
    <*fmt.wrapError | 0xc00110cee0>: 
    query graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:257]: failed to get version pair without upgrade edge for minor 4.20
Unexpected error:
    <*fmt.wrapError | 0xc00110cee0>: 
    query graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    ...
occurred
failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 503 Service Unavailable: <html>
07:08:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 504 Gateway Timeout: upstream request timeout
07:08:34 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 504 Gateway Timeout: upstream request timeout
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 504 Gateway Timeout: upstream request timeout
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
07:08:34 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc002402bd0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-jdfzts, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-jdfzts" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc002402bd0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-jdfzts, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-jdfzts" to finish creating: context deadline exceeded
    ...
occurred
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
07:08:34 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:400]: failed to get latest version for nodepool minor 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0008085c0>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:400]: failed to get latest version for nodepool minor 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0008085c0>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
07:08:34 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d62860>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d62860>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
07:08:34 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:77]: failed to find version in minor 4.20 with upgrade path to 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000fc5380>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:77]: failed to find version in minor 4.20 with upgrade path to 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000fc5380>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
    ...
occurred
after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
07:08:34 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000c2e120>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000c2e120>: 
    after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
    ...
occurred
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
07:08:34 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 6
e2e (6)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:17 UTC · failed at e2e · test FIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should create an HCP cluster with FIPS mode enabled via experimental tag · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:100]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc001d83770>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group fips-enabled-b4728v, error: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b4728v" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:100]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc001d83770>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group fips-enabled-b4728v, error: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b4728v" to finish creating: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-qftg2g" for cluster "cluster-candidate-4-21-qftg2g"
Unexpected error:
    <*fmt.wrapErrors | 0xc000f98120>: 
    failed to create NodePool np-qftg2g, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-qftg2g in resource group rg-candidate-4-21-vrw5rj, error: failed waiting for nodepool="np-qftg2g" for cluster "cluster-candidate-4-21-qftg2g" in resourcegroup="rg-candidate-4-21-vrw5rj" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-qftg2g" for cluster "cluster-candidate-4-21-qftg2g"
Unexpected error:
    <*fmt.wrapErrors | 0xc000f98120>: 
    failed to create NodePool np-qftg2g, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-qftg2g in resource group rg-candidate-4-21-vrw5rj, error: failed waiting for nodepool="np-qftg2g" for cluster "cluster-candidate-4-21-qftg2g" in resourcegroup="rg-candidate-4-21-vrw5rj" to finish creating: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:17 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001d2c3a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-ppj4zt, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ppj4zt" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001d2c3a8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-ppj4zt, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ppj4zt" to finish creating: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:17 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0021aaf30>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-zb4rqc, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-zb4rqc" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0021aaf30>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-zb4rqc, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-zb4rqc" to finish creating: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
02:00:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-b7pgw7" for cluster "cluster-candidate-4-20-b7pgw7"
Unexpected error:
    <*fmt.wrapErrors | 0xc002923b00>: 
    failed to create NodePool np-b7pgw7, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-b7pgw7 in resource group rg-candidate-4-20-f85lwv, error: failed waiting for nodepool="np-b7pgw7" for cluster "cluster-candidate-4-20-b7pgw7" in resourcegroup="rg-candidate-4-20-f85lwv" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-b7pgw7" for cluster "cluster-candidate-4-20-b7pgw7"
Unexpected error:
    <*fmt.wrapErrors | 0xc002923b00>: 
    failed to create NodePool np-b7pgw7, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-b7pgw7 in resource group rg-candidate-4-20-f85lwv, error: failed waiting for nodepool="np-b7pgw7" for cluster "cluster-candidate-4-20-b7pgw7" in resourcegroup="rg-candidate-4-20-f85lwv" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
02:00:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-mchxx4/cluster-candidate-5-0-jn78nj should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0005be780>: 
    failed to create HCP cluster cluster-candidate-5-0-jn78nj, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-jn78nj in resource group rg-candidate-5-0-mchxx4, error: failed waiting for cluster="cluster-candidate-5-0-jn78nj" in resourcegroup="rg-candidate-5-0-mchxx4" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-jn78nj in resource group rg-candidate-5-0-mchxx4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-mchxx4/cluster-candidate-5-0-jn78nj should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0005be780>: 
    failed to create HCP cluster cluster-candidate-5-0-jn78nj, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-jn78nj in resource group rg-candidate-5-0-mchxx4, error: failed waiting for cluster="cluster-candidate-5-0-jn78nj" in resourcegroup="rg-candidate-5-0-mchxx4" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-jn78nj in resource group rg-candidate-5-0-mchxx4, error: context deadline exceeded
    ...
occurred