CIHealth
<
Single Day: Jun 10▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:00:01 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-9kjrzr at version 4.22.0
Unexpected error:
    <*fmt.wrapError | 0xc000e08900>: 
    failed to create HCP cluster np-skip-minor-9kjrzr: failed starting cluster creation "np-skip-minor-9kjrzr" in resourcegroup="rg-np-skip-minor-9kjrzr-7gs6rf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9kjrzr-7gs6rf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-9kjrzr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-9kjrzr at version 4.22.0
Unexpected error:
    <*fmt.wrapError | 0xc000e08900>: 
    failed to create HCP cluster np-skip-minor-9kjrzr: failed starting cluster creation "np-skip-minor-9kjrzr" in resourcegroup="rg-np-skip-minor-9kjrzr-7gs6rf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-9kjrzr-7gs6rf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-9kjrzr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.22.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v<version>' doesn't exist; provider Microsoft.RedHatOpenShift
23:00:01 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9caa0>: 
    failed to create HCP cluster np-dg-minor-9c7d7p: failed starting cluster creation "np-dg-minor-9c7d7p" in resourcegroup="rg-np-dg-minor-9c7d7p-rhs2s4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-9c7d7p-rhs2s4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-9c7d7p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:704]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9caa0>: 
    failed to create HCP cluster np-dg-minor-9c7d7p: failed starting cluster creation "np-dg-minor-9c7d7p" in resourcegroup="rg-np-dg-minor-9c7d7p-rhs2s4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-9c7d7p-rhs2s4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-9c7d7p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.21.0-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
19:22:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-c7x88v/cluster-nightly-5-0-msgf8g should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00053a0f0>: 
    failed waiting for cluster="cluster-nightly-5-0-msgf8g" in resourcegroup="rg-nightly-5-0-c7x88v" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-msgf8g in resource group rg-nightly-5-0-c7x88v, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-c7x88v/cluster-nightly-5-0-msgf8g should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc00053a0f0>: 
    failed waiting for cluster="cluster-nightly-5-0-msgf8g" in resourcegroup="rg-nightly-5-0-c7x88v" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-msgf8g in resource group rg-nightly-5-0-c7x88v, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
Interrupted by User
06:43:36 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Interrupted by User
06:43:36 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
timeout during CreateHCPClusterAndWait; context deadline exceeded
06:42:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-kcn82j/cluster-nightly-5-0-5r4lh5 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001322450>: 
    failed waiting for cluster="cluster-nightly-5-0-5r4lh5" in resourcegroup="rg-nightly-5-0-kcn82j" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-5r4lh5 in resource group rg-nightly-5-0-kcn82j, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-kcn82j/cluster-nightly-5-0-5r4lh5 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001322450>: 
    failed waiting for cluster="cluster-nightly-5-0-5r4lh5" in resourcegroup="rg-nightly-5-0-kcn82j" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-5r4lh5 in resource group rg-nightly-5-0-kcn82j, error: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
06:42:17 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8"
Unexpected error:
    <*fmt.wrapErrors | 0xc00083e120>: 
    failed to create NodePool np-jm8qc8, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-jm8qc8 in resource group rg-candidate-4-20-svkpkx, error: failed waiting for nodepool="np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8" in resourcegroup="rg-candidate-4-20-svkpkx" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:192]: failed to create node pool "np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8"
Unexpected error:
    <*fmt.wrapErrors | 0xc00083e120>: 
    failed to create NodePool np-jm8qc8, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-jm8qc8 in resource group rg-candidate-4-20-svkpkx, error: failed waiting for nodepool="np-jm8qc8" for cluster "cluster-candidate-4-20-jm8qc8" in resourcegroup="rg-candidate-4-20-svkpkx" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
06:41:04 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00120e3f0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-nlck2h, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-nlck2h" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00120e3f0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-nlck2h, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-nlck2h" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
06:41:04 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-5xwj2c/cluster-nightly-5-0-9hzz55 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0005304b0>: 
    failed waiting for cluster="cluster-nightly-5-0-9hzz55" in resourcegroup="rg-nightly-5-0-5xwj2c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-9hzz55 in resource group rg-nightly-5-0-5xwj2c, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-5-0-5xwj2c/cluster-nightly-5-0-9hzz55 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0005304b0>: 
    failed waiting for cluster="cluster-nightly-5-0-9hzz55" in resourcegroup="rg-nightly-5-0-5xwj2c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cluster-nightly-5-0-9hzz55 in resource group rg-nightly-5-0-5xwj2c, error: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.19.33": cannot downgrade from version 4.21.19 to 4.19.33; provider Microsoft.RedHatOpenShift
02:00:04 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc000d0e180>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-hf2g5m" resourcegroup="rg-np-dg-minor-hf2g5m-8ws4v8": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hf2g5m-8ws4v8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hf2g5m/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:750]: failed to update nodepool npdg-421-419 to downgrade target 4.19.33
Unexpected error:
    <*fmt.wrapError | 0xc000d0e180>: 
    failed to start nodepool "npdg-421-419" update in cluster "np-dg-minor-hf2g5m" resourcegroup="rg-np-dg-minor-hf2g5m-8ws4v8": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hf2g5m-8ws4v8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hf2g5m/nodePools/npdg-421-419
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": invalid node pool version 4.19.33: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.19.33\": cannot downgrade from version 4.21.19 to 4.19.33",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21.18": cannot downgrade from version 4.21.19 to 4.21.18; provider Microsoft.RedHatOpenShift
02:00:04 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
    <*fmt.wrapError | 0xc000dc8ea0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-62m6dt" resourcegroup="rg-np-downgrade-62m6dt-z875wp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-62m6dt-z875wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-62m6dt/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:619]: failed to update nodepool npdg-4-21 to downgrade target 4.21.18
Unexpected error:
    <*fmt.wrapError | 0xc000dc8ea0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-62m6dt" resourcegroup="rg-np-downgrade-62m6dt-z875wp": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-62m6dt-z875wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-62m6dt/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": invalid node pool version 4.21.18: cannot downgrade from current version 4.21.19",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21.18\": cannot downgrade from version 4.21.19 to 4.21.18",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred