CIHealth
<
Single Day: Sep 30▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: INT

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 3m 5s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 e2e failed n/a 11
e2e (11)
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-w4nsqz" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000d2e780>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nhrfp5qbdv2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-w4nsqz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-w4nsqz" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000d2e780>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nhrfp5qbdv2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-w4nsqz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-lr7n9c" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00144e180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-zh65jflwpg9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-lr7n9c
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-lr7n9c" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00144e180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-zh65jflwpg9j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-lr7n9c
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-nkn9cp" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000fb60c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2s89ppfpwxxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-nkn9cp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-nkn9cp" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000fb60c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2s89ppfpwxxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-nkn9cp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-8g7jd8" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014b4e10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-c244qsxrhnbn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-8g7jd8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-8g7jd8" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014b4e10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-c244qsxrhnbn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-8g7jd8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
22:21:45 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d6f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-t8kp5gffh7ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e",
      "name": "fbb7c321-b5c6-4145-9be8-48a64282647e",
      "status": "Failed",
      "startTime": "2026-09-30T22:47:02.188860165Z",
      "endTime": "2026-09-30T23:06:03.834732494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d6f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-t8kp5gffh7ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e",
      "name": "fbb7c321-b5c6-4145-9be8-48a64282647e",
      "status": "Failed",
      "startTime": "2026-09-30T22:47:02.188860165Z",
      "endTime": "2026-09-30T23:06:03.834732494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-52n697" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0015481e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-52jx5pj75l4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-52n697
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-52n697" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0015481e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-52jx5pj75l4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-52n697
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-j4z2m9" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000c73320>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-xlhwbtlldbgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-j4z2m9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-j4z2m9" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000c73320>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-xlhwbtlldbgk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-j4z2m9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-b9ld4k" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000a93bf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-m7g7x4rhlkmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-b9ld4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-b9ld4k" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000a93bf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-m7g7x4rhlkmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-b9ld4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-6czr45" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00106e2a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zs8jz58nhzb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-6czr45
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-6czr45" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00106e2a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zs8jz58nhzb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-6czr45
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d9vb4z" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000f5c990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-v5vn6xvq7g9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-d9vb4z
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d9vb4z" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000f5c990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-v5vn6xvq7g9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-d9vb4z
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
22:21:45 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xw9s99" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00095f590>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-248rf82xp8ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xw9s99
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-xw9s99" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00095f590>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-248rf82xp8ms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xw9s99
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 7m 5s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:11:51 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc001088480>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-mmgvqz" in resourcegroup="rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846",
      "name": "b1caf488-0088-4fa0-8a37-c1f30126d846",
      "status": "Failed",
      "startTime": "2026-09-30T20:37:05.655658972Z",
      "endTime": "2026-09-30T20:56:13.566012243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc001088480>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-mmgvqz" in resourcegroup="rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846",
      "name": "b1caf488-0088-4fa0-8a37-c1f30126d846",
      "status": "Failed",
      "startTime": "2026-09-30T20:37:05.655658972Z",
      "endTime": "2026-09-30T20:56:13.566012243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 25m 34s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 e2e failed n/a 1
e2e (1)
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>
16:37:51 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-20-t74pss"
Unexpected error:
    <*fmt.wrapError | 0xc000b40260>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-t74pss-7wvjkj" in resourcegroup="rg-candidate-4-20-bmfs7fmklpfb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-bmfs7fmklpfb/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-t74pss-7wvjkj/operationStatuses/08584108180410013833
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-cr5ick5opsuzu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-20-t74pss"
Unexpected error:
    <*fmt.wrapError | 0xc000b40260>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-t74pss-7wvjkj" in resourcegroup="rg-candidate-4-20-bmfs7fmklpfb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-bmfs7fmklpfb/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-t74pss-7wvjkj/operationStatuses/08584108180410013833
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-cr5ick5opsuzu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
22s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
9m 14s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
2h 36m 4s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 n/a passed n/a 0 n/a
2h 29m 1s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 n/a passed n/a 0 n/a
2h 24m 13s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 n/a passed n/a 0 n/a