CIHealth
<
Single Day: Oct 1▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: INT

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 22m 46s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 n/a passed n/a 0 n/a
2h 30m 55s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 e2e failed n/a 2
e2e (2)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:47:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk"
Unexpected error:
    <*fmt.wrapError | 0xc000941160>: 
    failed to create NodePool np-jv9hvk: failed waiting for nodepool="np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk" in resourcegroup="rg-candidate-4-20-6nfp6mj4gt68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "name": "29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "status": "Failed",
      "startTime": "2026-10-01T20:46:24.926645724Z",
      "endTime": "2026-10-01T21:05:28.967720321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk"
Unexpected error:
    <*fmt.wrapError | 0xc000941160>: 
    failed to create NodePool np-jv9hvk: failed waiting for nodepool="np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk" in resourcegroup="rg-candidate-4-20-6nfp6mj4gt68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "name": "29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "status": "Failed",
      "startTime": "2026-10-01T20:46:24.926645724Z",
      "endTime": "2026-10-01T21:05:28.967720321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
19:47:05 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012c9e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fxmrv96q875w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269",
      "name": "a1048b90-244a-4c5a-ba89-e1135a68a269",
      "status": "Failed",
      "startTime": "2026-10-01T20:23:38.274618399Z",
      "endTime": "2026-10-01T20:42:38.037012934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012c9e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fxmrv96q875w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269",
      "name": "a1048b90-244a-4c5a-ba89-e1135a68a269",
      "status": "Failed",
      "startTime": "2026-10-01T20:23:38.274618399Z",
      "endTime": "2026-10-01T20:42:38.037012934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 23m 26s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 n/a passed n/a 0 n/a
2h 16m 53s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 e2e failed n/a 10
e2e (10)
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-282pdp" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000fe7f50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-qszf5tc8smsd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-282pdp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-282pdp" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000fe7f50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-qszf5tc8smsd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-282pdp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-fpkhgg" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0013695c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-x2qm64ntnf4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-fpkhgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-fpkhgg" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0013695c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-x2qm64ntnf4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-fpkhgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-zhcqgm" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000f2be90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-bkxc6qpdr2kc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-zhcqgm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-zhcqgm" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000f2be90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-bkxc6qpdr2kc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-zhcqgm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-ncq7tv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0014ac6f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jnrgf69jtww7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-ncq7tv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-ncq7tv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0014ac6f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jnrgf69jtww7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-ncq7tv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5jgcvl" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00156e510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-qqh9mh2dgv5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5jgcvl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5jgcvl" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00156e510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-qqh9mh2dgv5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5jgcvl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-cznf64" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0018c4c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-z6bx6clhw9nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-cznf64
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-cznf64" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0018c4c30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-z6bx6clhw9nr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-cznf64
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5lppxg" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c77c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-9qkw2hc8j64k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5lppxg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5lppxg" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c77c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-9qkw2hc8j64k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5lppxg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-rtvfwf" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0012da990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zzffnqvsn5bk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-rtvfwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-rtvfwf" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0012da990>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zzffnqvsn5bk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-rtvfwf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-59lhdf" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b51650>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-zncx9w9pdbrq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-59lhdf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-59lhdf" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b51650>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-zncx9w9pdbrq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-59lhdf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
10:32:01 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-5gft95" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0015d78c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-8c7gqffnfmhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-5gft95
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-5gft95" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0015d78c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-8c7gqffnfmhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-5gft95
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 6m 46s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 e2e failed n/a 10
e2e (10)
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-tvg8ls" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001483050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-8ptjnkq2z7df/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-tvg8ls
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-tvg8ls" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001483050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-8ptjnkq2z7df/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-tvg8ls
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fbsgr5" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00153e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rfwzkqxm7bss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-fbsgr5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fbsgr5" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00153e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rfwzkqxm7bss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-fbsgr5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-gfc5ks" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b4b2f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2k4s9c5c5gt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-gfc5ks
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-gfc5ks" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000b4b2f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2k4s9c5c5gt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-gfc5ks
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-skgz2q" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0014816b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-j4gwfrfwllb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-skgz2q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-skgz2q" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0014816b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-j4gwfrfwllb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-skgz2q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-97ngnf" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0009a0ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-l7xhqzrtl9sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-97ngnf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-97ngnf" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0009a0ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-l7xhqzrtl9sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-97ngnf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-84kjqn" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00159a510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-9tsgcnv7kf56/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-84kjqn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-84kjqn" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00159a510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-9tsgcnv7kf56/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-84kjqn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-6vqlpz" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001492180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-6954mm5k46dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-6vqlpz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-6vqlpz" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001492180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-6954mm5k46dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-6vqlpz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-gkw8ql" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0011fba70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-lzvgwqsshhgh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-gkw8ql
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-gkw8ql" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0011fba70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-lzvgwqsshhgh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-gkw8ql
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qkh6np" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000af7500>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-jqv2lkbmlggl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qkh6np
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qkh6np" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000af7500>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-jqv2lkbmlggl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qkh6np
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
07:19:36 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5xch77" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001152bd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-869wg248knm4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5xch77
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5xch77" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001152bd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-869wg248knm4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5xch77
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 7m 51s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 westus3 e2e failed n/a 12
e2e (12)
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-rngq5s" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000ccdb30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-mlc5mpzcvj8q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rngq5s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-rngq5s" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc000ccdb30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-mlc5mpzcvj8q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rngq5s
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-zx7mxd" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0018b1e90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gvghfmdnj9kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-zx7mxd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-zx7mxd" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0018b1e90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gvghfmdnj9kp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-zx7mxd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-tkvcqf" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00155e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-9l9l89pgr4m5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-tkvcqf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-tkvcqf" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00155e240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-9l9l89pgr4m5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-tkvcqf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-4p7wkl" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc001587980>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2dqnlv5bkbch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-4p7wkl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-4p7wkl" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc001587980>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-2dqnlv5bkbch/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-4p7wkl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00109d0c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s7p6hptqhbvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1",
      "name": "357a31b9-a87e-4b62-a8b5-196b066234f1",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:38.669148771Z",
      "endTime": "2026-10-01T03:17:44.618360897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00109d0c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s7p6hptqhbvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1",
      "name": "357a31b9-a87e-4b62-a8b5-196b066234f1",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:38.669148771Z",
      "endTime": "2026-10-01T03:17:44.618360897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-tsd828" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000ab0180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-qpv6d29gz4jl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-tsd828
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-tsd828" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000ab0180>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-qpv6d29gz4jl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-tsd828
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-q5mklf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00118f7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-rcvrxff9wvxp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-q5mklf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-q5mklf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00118f7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-rcvrxff9wvxp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-q5mklf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-gswqqb" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0005f0a80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-nmmbsqvg94ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gswqqb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-gswqqb" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0005f0a80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-nmmbsqvg94ml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gswqqb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-vnllsd" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0015a4150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-5lmngc5rw784/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-vnllsd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-vnllsd" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc0015a4150>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-5lmngc5rw784/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-vnllsd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
01:33:40 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00057dcc0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jspphct49kmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb",
      "name": "4a191606-d1f9-418e-b590-0307f5435adb",
      "status": "Failed",
      "startTime": "2026-10-01T01:58:13.545499376Z",
      "endTime": "2026-10-01T02:17:18.498756095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00057dcc0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jspphct49kmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb",
      "name": "4a191606-d1f9-418e-b590-0307f5435adb",
      "status": "Failed",
      "startTime": "2026-10-01T01:58:13.545499376Z",
      "endTime": "2026-10-01T02:17:18.498756095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-2crl7w" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0015c2c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-bdf2955zlsb2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2crl7w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-2crl7w" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0015c2c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-bdf2955zlsb2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-2crl7w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
01:33:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4jz6tz" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000a1c2d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wjhccp827grc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4jz6tz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4jz6tz" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000a1c2d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wjhccp827grc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4jz6tz
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred