CIHealth
<
Single Day: Oct 10▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 4m 42s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2654cb9 · merge 2654cb9
 westus3 n/a passed #7374 (merged) 0 n/a
2h 17m 48s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head de99ea9 · merge de99ea9
 centralus n/a passed #7456 (merged) 0 n/a
2h 49m 25s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 349d3b8 · merge 349d3b8
 westus3 n/a passed #7227 (merged) 0 n/a
2h 20m 39s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2654cb9 · merge 2654cb9
 westus3 alert, e2e failed ★#7374 (merged) 2
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
11:45:56 UTC · failed at e2e · test Nodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x2179351f8800>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-5d4jlxd268w8" to finish creating: GET https://rp.j0635008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0",
      "name": "5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0",
      "status": "Failed",
      "startTime": "2026-10-10T12:54:39.340662434Z",
      "endTime": "2026-10-10T12:55:18.027112085Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-5d4jlxd268w8/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0635008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0
            correlation ID: 13bc1f0a-5221-4912-b516-aa03e8a99e61
            client request ID: 5d284363-4eac-4367-9c73-8148dde5d509
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x2179351f8800>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-5d4jlxd268w8" to finish creating: GET https://rp.j0635008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0",
      "name": "5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0",
      "status": "Failed",
      "startTime": "2026-10-10T12:54:39.340662434Z",
      "endTime": "2026-10-10T12:55:18.027112085Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-5d4jlxd268w8/providers/Microsoft.Compute/diskEncryptionSets/des-encrypt-des\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0635008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a75eacc-f787-4eeb-91c6-ba37f2f3c7e0
            correlation ID: 13bc1f0a-5221-4912-b516-aa03e8a99e61
            client request ID: 5d284363-4eac-4367-9c73-8148dde5d509
occurred
alert (1)
alert [svc] LeaderElectionLeaseStale fired
11:45:56 UTC · failed at alert · test [aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T13:49:18Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j0635008-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.197:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0635008/providers/microsoft.monitor/accounts/services-j0635008", namespace="klusterlet-2tbu9f66u76ktrmgh418ji9vp03p5kjp", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8w74x", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tbu9f66u76ktrmgh418ji9vp03p5kjp on cluster ci01-j0635008-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
2h 7m 50s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head de99ea9 · merge de99ea9
 westus3 n/a passed #7456 (merged) 0 n/a
2h 18m 24s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 349d3b8 · merge 349d3b8
 westus3 e2e failed ★#7227 (merged) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
08:03:23 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x5b5e959eb80>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hxz5lhgx6pgc" to finish creating: GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "name": "bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "status": "Failed",
      "startTime": "2026-10-10T09:08:14.887674935Z",
      "endTime": "2026-10-10T09:37:18.731959559Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
            correlation ID: 85344fe9-db3b-408b-ba74-a3aed4f6938b
            client request ID: 900c9a16-ab81-46bf-aba5-3f67f267761a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x5b5e959eb80>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hxz5lhgx6pgc" to finish creating: GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "name": "bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f",
      "status": "Failed",
      "startTime": "2026-10-10T09:08:14.887674935Z",
      "endTime": "2026-10-10T09:37:18.731959559Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8557440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bd8eab11-2b4f-4ee5-bf79-f2c3f0cf511f
            correlation ID: 85344fe9-db3b-408b-ba74-a3aed4f6938b
            client request ID: 900c9a16-ab81-46bf-aba5-3f67f267761a
occurred
2h 26m 53s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2654cb9 · merge 2654cb9
 westus3 alert failed #7374 (merged) 1
alert (1)
alert [hcp] KubePodNotReady fired
07:19:42 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-10T08:50:24Z
  Ended:       2026-10-10T08:56:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9837312-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbqdq7nkko6nfm8vch1p3aq46onal67-private-kas", pod="cluster-policy-controller-86bf79cf56-rh2l8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbqdq7nkko6nfm8vch1p3aq46onal67-private-kas/cluster-policy-controller-86bf79cf56-rh2l8 has been in a non-ready state for longer than 5 minutes.
2h 38m 57s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 96338d0 · merge 96338d0
 westus3 n/a passed #7165 (merged) 0 n/a
2h 17m 6s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head ff34e72
 westus3 n/a passed #7340 (open) 0 n/a
2h 26m 5s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 982ccea
 centralus n/a passed #7440 (open) 0 n/a
2h 22m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 349d3b8 · merge 349d3b8
 centralus e2e failed ★#7227 (merged) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-drwbc-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-drwbc-<id> to exist; provider Microsoft.RedHatOpenShift
05:02:46 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x24a46d574920>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlxq4qgj7xpp" to finish creating: GET https://rp.j0639104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/309223c5-c91d-4270-b774-50dec66e2570
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/309223c5-c91d-4270-b774-50dec66e2570",
      "name": "309223c5-c91d-4270-b774-50dec66e2570",
      "status": "Failed",
      "startTime": "2026-10-10T06:15:46.054292016Z",
      "endTime": "2026-10-10T06:44:53.542490042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-drwbc-mbw6z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mlxq4qgj7xpp--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-drwbc-mbw6z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-drwbc-mbw6z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mlxq4qgj7xpp--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-drwbc-mbw6z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0639104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/309223c5-c91d-4270-b774-50dec66e2570
            correlation ID: 57417513-291c-4d90-a0bc-a093da0940a4
            client request ID: 3e97b4e2-ed3b-4f11-be4f-69ec5d075229
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x24a46d574920>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlxq4qgj7xpp" to finish creating: GET https://rp.j0639104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/309223c5-c91d-4270-b774-50dec66e2570
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/309223c5-c91d-4270-b774-50dec66e2570",
      "name": "309223c5-c91d-4270-b774-50dec66e2570",
      "status": "Failed",
      "startTime": "2026-10-10T06:15:46.054292016Z",
      "endTime": "2026-10-10T06:44:53.542490042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-drwbc-mbw6z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mlxq4qgj7xpp--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-drwbc-mbw6z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-drwbc-mbw6z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mlxq4qgj7xpp--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-drwbc-mbw6z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0639104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/309223c5-c91d-4270-b774-50dec66e2570
            correlation ID: 57417513-291c-4d90-a0bc-a093da0940a4
            client request ID: 3e97b4e2-ed3b-4f11-be4f-69ec5d075229
occurred
2h 26m 45s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head cb92dba
 centralus e2e failed #7439 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:02:26 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x23a798e4e3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7kr9hzf26ffd" to finish creating: GET https://rp.j0742656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357",
      "name": "2a131d64-b48f-453b-b60d-f4063709a357",
      "status": "Failed",
      "startTime": "2026-10-10T03:18:06.524337179Z",
      "endTime": "2026-10-10T03:47:07.577893958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x23a798e4e3a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7kr9hzf26ffd" to finish creating: GET https://rp.j0742656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a131d64-b48f-453b-b60d-f4063709a357",
      "name": "2a131d64-b48f-453b-b60d-f4063709a357",
      "status": "Failed",
      "startTime": "2026-10-10T03:18:06.524337179Z",
      "endTime": "2026-10-10T03:47:07.577893958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-nk7dk to exist\nMachine p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kr9hzf26ffd--managed/providers/Microsoft.Compute/virtualMachines/p9a2k4m8s9w5v9a-np-1-b4xfk-vck9z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 8m 27s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 868b002
 westus3 e2e failed #7177 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
01:21:53 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x560689fa080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-jzlxwvqzg694" to finish creating: GET https://rp.j5842176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "name": "efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "status": "Failed",
      "startTime": "2026-10-10T02:25:01.899620825Z",
      "endTime": "2026-10-10T02:54:06.245842706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x560689fa080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-jzlxwvqzg694" to finish creating: GET https://rp.j5842176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "name": "efe266bd-17ba-4b43-bc21-b2099c2f07cf",
      "status": "Failed",
      "startTime": "2026-10-10T02:25:01.899620825Z",
      "endTime": "2026-10-10T02:54:06.245842706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6u6d5o1h9k6y0v-np-1-wpskn-72d9j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jzlxwvqzg694--managed/providers/Microsoft.Compute/virtualMachines/j6u6d5o1h9k6y0v-np-1-wpskn-72d9j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 13m 20s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 96338d0 · merge 96338d0
 centralus e2e failed #7165 (merged) 1
e2e (1)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
00:06:26 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x7f83ee01c00>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c4ck5gxmvgl2" to finish creating: GET https://rp.j2487424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "name": "b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "status": "Failed",
      "startTime": "2026-10-10T01:21:08.093175992Z",
      "endTime": "2026-10-10T01:50:13.893881459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x7f83ee01c00>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-c4ck5gxmvgl2" to finish creating: GET https://rp.j2487424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "name": "b0e10a40-6968-4b3e-9cdc-0302b36aa6cc",
      "status": "Failed",
      "startTime": "2026-10-10T01:21:08.093175992Z",
      "endTime": "2026-10-10T01:50:13.893881459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-c4ck5gxmvgl2--managed/providers/Microsoft.Compute/virtualMachines/m6u2s5a2u3w6s5j-np-delete-main-skm2q-qflrc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 47m 30s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 canadacentral e2e failed n/a 6
e2e (6)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:53 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x35e890969710>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vwbbqk8fgxxg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vwbbqk8fgxxg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: afc2359c-ef40-4afe-936e-d1b3ca33a454
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x35e890969710>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vwbbqk8fgxxg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vwbbqk8fgxxg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: afc2359c-ef40-4afe-936e-d1b3ca33a454
            client request ID: unavailable
occurred
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
23:00:53 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x3ac58974db78>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3ac589d3d2a0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589d3d280>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3ac589190be0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3ac589e5f3e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589e5f3c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3ac58a59a640>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:98]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
    <*errors.joinError | 0x3ac58974db78>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x3ac589d3d2a0>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589d3d280>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x3ac589190be0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x3ac589e5f3e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x3ac589e5f3c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x3ac58a59a640>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:53 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x27c61870b710>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-26r9262rnlqd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-26r9262rnlqd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: a6f275cb-a77b-4935-a20b-72bac7fef063
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x27c61870b710>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-26r9262rnlqd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-26r9262rnlqd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: a6f275cb-a77b-4935-a20b-72bac7fef063
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:53 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x7c58726bb30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xh7vmfjmkg99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xh7vmfjmkg99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: fe3817e1-9407-4f34-956f-bce48e1f6d2d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x7c58726bb30>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xh7vmfjmkg99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-xh7vmfjmkg99/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: fe3817e1-9407-4f34-956f-bce48e1f6d2d
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:53 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x3bba69c1ca80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rr7hzrxzk6fn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rr7hzrxzk6fn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 60531111-8f02-4e39-81fb-9b39df575394
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x3bba69c1ca80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rr7hzrxzk6fn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-rr7hzrxzk6fn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: 60531111-8f02-4e39-81fb-9b39df575394
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:53 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x27a78177f200>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-zdpr2w926sdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-zdpr2w926sdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 57d63011-6fd4-4fc5-b033-1533b532503a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x27a78177f200>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-zdpr2w926sdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-zdpr2w926sdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 57d63011-6fd4-4fc5-b033-1533b532503a
            client request ID: unavailable
occurred
2h 20m 38s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast n/a passed n/a 0 n/a

Environment: STG

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 25m 59s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 5
e2e (5)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:54 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x2601f3682270>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-w4p8zgstrslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-w4p8zgstrslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: 452c080d-fbc6-4f24-b6c7-c80a5ac391c6
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x2601f3682270>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-w4p8zgstrslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-w4p8zgstrslb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
            correlation ID: 452c080d-fbc6-4f24-b6c7-c80a5ac391c6
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:54 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x201deabbb140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qpbqdblcz7mt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qpbqdblcz7mt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 449762fe-87a7-4cf8-8168-d2937d6d5733
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x201deabbb140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qpbqdblcz7mt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qpbqdblcz7mt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
            correlation ID: 449762fe-87a7-4cf8-8168-d2937d6d5733
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:54 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x18454626b020>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxgnzb9gtg4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxgnzb9gtg4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 83dca14d-2dc8-41de-990f-d29b19bed06d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x18454626b020>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxgnzb9gtg4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxgnzb9gtg4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
            correlation ID: 83dca14d-2dc8-41de-990f-d29b19bed06d
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:54 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2e1fdc8a0c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-p8t2w8csmrsx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-p8t2w8csmrsx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: cedccd74-e12e-4277-aa61-796ff6f79e25
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2e1fdc8a0c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-p8t2w8csmrsx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-p8t2w8csmrsx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
            correlation ID: cedccd74-e12e-4277-aa61-796ff6f79e25
            client request ID: unavailable
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:54 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x21fd4af257d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-b2jxktqft79k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-b2jxktqft79k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: a6fa2e81-3c9c-4b8f-8881-894ce6c122fe
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x21fd4af257d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-b2jxktqft79k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-b2jxktqft79k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
            correlation ID: a6fa2e81-3c9c-4b8f-8881-894ce6c122fe
            client request ID: unavailable
occurred
2h 6m 10s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a