CIHealth
<
Last 7 Days▾
Last 7 Days Weekly: Aug 24 - Aug 30 Sprint: Aug 17 - Aug 30 Single Day: Aug 24
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 166 · Failures matched: 1388 · Total runs: 587 · Runs affected: 390

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.17%1 of 587 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00133a200>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-qp6jkgrm7jtk" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09",
      "name": "7d814403-8a1a-482d-afa5-63e68ef4ed09",
      "status": "Failed",
      "startTime": "2026-08-24T21:13:13.654576687Z",
      "endTime": "2026-08-24T21:32:17.358471293Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00133a200>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-qp6jkgrm7jtk" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7d814403-8a1a-482d-afa5-63e68ef4ed09",
      "name": "7d814403-8a1a-482d-afa5-63e68ef4ed09",
      "status": "Failed",
      "startTime": "2026-08-24T21:13:13.654576687Z",
      "endTime": "2026-08-24T21:32:17.358471293Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0012a4ea0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-26mdm7dwkvm7" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9",
      "name": "fea58607-6b10-431a-a2a1-cc0c116e10b9",
      "status": "Failed",
      "startTime": "2026-08-24T21:12:09.791388432Z",
      "endTime": "2026-08-24T21:31:17.457093893Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0012a4ea0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-26mdm7dwkvm7" to finish creating: GET https://rp.j8136960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fea58607-6b10-431a-a2a1-cc0c116e10b9",
      "name": "fea58607-6b10-431a-a2a1-cc0c116e10b9",
      "status": "Failed",
      "startTime": "2026-08-24T21:12:09.791388432Z",
      "endTime": "2026-08-24T21:31:17.457093893Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #66412091984065988136960
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code StorageAccountOperationInProgress; detail message An operation is currently performing on this storage account that requires exclusive ac...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code StorageAccountOperationInProgress; detail message An operation is currently performing on this storage account that requires exclusive access.
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.17%1 of 587 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
time=2026-08-24T16:09:49.176Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 description="Step infra\n  Kind: ARM\n  Template: templates/mgmt-infra.bicep\n  Parameters: configurations/mgmt-infra.tmpl.bicepparam"
time=2026-08-24T16:09:49.840Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2
time=2026-08-24T16:09:50.553Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 deployment=c783e3e3d697ee4d22dffbab4271479e105f154955a91c3a47e9819df57ac290 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1968512-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc783e3e3d697ee4d22dffbab4271479e105f154955a91c3a47e9819df57ac290
time=2026-08-24T16:11:20.899Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/c783e3e3d697ee4d22dffbab4271479e105f154955a91c3a47e9819df57ac290/operationStatuses/08584140190952500816\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/hcp-backups-storage\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/hcpBackupsStorageAccount\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1968512-mgmt-2/providers/Microsoft.Resources/deployments/hcpBackupsStorageAccount\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"StorageAccountOperationInProgress\\\",\\r\\n                    \\\"message\\\": \\\"An operation is currently performing on this storage account that requires exclusive access.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/infra (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #66592091914307531968512
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGrid
full failure pattern:
ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGrid
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.17%1 of 587 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
time=2026-08-24T07:13:06.981Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-08-24T07:13:09.018Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-08-24T07:13:10.536Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9665ebaf931e927bcf7f137293ddd5a9262983f1f2075a0dff43a22749c6dc59 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1860736%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9665ebaf931e927bcf7f137293ddd5a9262983f1f2075a0dff43a22749c6dc59
time=2026-08-24T07:29:44.382Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1860736/providers/Microsoft.Resources/deployments/9665ebaf931e927bcf7f137293ddd5a9262983f1f2075a0dff43a22749c6dc59/operationStatuses/08584140512958847265\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1860736/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1860736/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1860736\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Timeout\\\",\\r\\n                \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #62782091780518591860736
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentStackDeploymentFailed; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeploymentStackDeploymentFailed; provider Microsoft.Compute
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.17%1 of 587 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
time=2026-08-24T08:15:25.803Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nodepools stamp=2 description="Step nodepools\n  Kind: ARMStack\n  Template: templates/aks-nodepools.bicep\n  Parameters: configurations/mgmt-cluster-nodepools.tmpl.bicepparam"
time=2026-08-24T08:18:16.150Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nodepools stamp=2 err="stamp 2: failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/67d8fd4b-85de-4d57-a829-7ce0bf268c5d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/67d8fd4b-85de-4d57-a829-7ce0bf268c5d\",\n  \"name\": \"67d8fd4b-85de-4d57-a829-7ce0bf268c5d\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: '676c3fe4-b0a1-44f5-8602-c3c05012b379'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9428224-mgmt-2/providers/Microsoft.Resources/deployments/l4mv3scto7z6edigthkjdr62jbdrgoxoif33nvakxlwvg-260824084ipd0\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"OverconstrainedZonalAllocationRequest\",\n            \"message\": \"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9428224-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-41975975-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 3, 2), 'Standard_E8-4s_v6' (zones 3, 2), 'Standard_E8-4ds_v6' (zones 2, 3) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9428224-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-infra1-41975975-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nodepools (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #64472091793135989428224
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.17%1 of 587 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ef90e0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-2zg88lxq9q4w" to finish creating: GET https://rp.j3558656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317",
      "name": "b976f9f3-2891-4433-868a-76a8d23de317",
      "status": "Failed",
      "startTime": "2026-08-24T16:54:08.369328068Z",
      "endTime": "2026-08-24T17:13:08.208412685Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ef90e0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-2zg88lxq9q4w" to finish creating: GET https://rp.j3558656.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b976f9f3-2891-4433-868a-76a8d23de317",
      "name": "b976f9f3-2891-4433-868a-76a8d23de317",
      "status": "Failed",
      "startTime": "2026-08-24T16:54:08.369328068Z",
      "endTime": "2026-08-24T17:13:08.208412685Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE can pause schedules to stop backup execution for an HCP cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #65842091920407983558656
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRrun command "e2e-runcommand-1787582391625387873" failed with exit code 1 on VM "<vm>" (output: "Client Version: v1.36.4 Kustomize Version: v5.8.1")
full failure pattern:
run command "e2e-runcommand-1787582391625387873" failed with exit code 1 on VM "<vm>" (output: "Client Version: v1.36.4 Kustomize Version: v5.8.1")
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.17%1 of 587 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:160]: kubectl version should succeed from VM via private KAS internal LB (output: )
Unexpected error:
    <*errors.errorString | 0xc0007fbf60>: 
    run command "e2e-runcommand-1787582391625387873" failed with exit code 1 on VM "private-kas-test-vm" (output: "Client Version: v1.36.4\nKustomize Version: v5.8.1")
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:160]: kubectl version should succeed from VM via private KAS internal LB (output: )
Unexpected error:
    <*errors.errorString | 0xc0007fbf60>: 
    run command "e2e-runcommand-1787582391625387873" failed with exit code 1 on VM "private-kas-test-vm" (output: "Client Version: v1.36.4\nKustomize Version: v5.8.1")
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #64292091879954055696384
alert [svc] FrontendPathLatency fired
full failure pattern:
alert [svc] FrontendPathLatency fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
alert10117.21%101 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsAug 17: 3 · Aug 18: 21 · Aug 19: 23 · Aug 20: 33 · Aug 21: 24 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-21T19:35:28Z
  Ended:       2026-08-21T19:44:25Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9105664-svc", component="frontend", method="get", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/externalauths/{externalauthname}", severity="warning"
  Description: The 99th percentile of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/externalauths/{externalauthname} has exceeded 1 second over the past 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-21T19:36:27Z
  Ended:       2026-08-21T19:45:26Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9105664-svc", component="frontend", method="delete", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The 99th percentile of frontend request latency for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 1 second over the past 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-08-21T19:36:27Z
  Ended:       2026-08-21T19:43:30Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9105664-svc", component="frontend", method="get", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: The 99th percentile of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} has exceeded 1 second over the past 5 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-21T19:38:31Z
  Ended:       2026-08-21T19:46:29Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j3141504-svc", component="frontend", method="get", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The 99th percentile of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 1 second over the past 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-21T19:39:30Z
  Ended:       2026-08-21T19:47:30Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j3141504-svc", component="frontend", method="get", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: The 99th percentile of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters has exceeded 1 second over the past 5 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-21T17:10:50Z
  Ended:       2026-08-21T17:18:48Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j1918464-svc", component="frontend", method="put", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The 99th percentile of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 1 second over the past 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-21T17:27:51Z
  Ended:       2026-08-21T17:33:49Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j1918464-svc", component="frontend", method="post", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential", severity="warning"
  Description: The 99th percentile of frontend request latency for post /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential has exceeded 1 second over the past 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire101
Affected runs (101)
Date (UTC)Associated PRProw job
PR #66522090868362149105664
PR #66502090866482933141504
PR #66312090837156841918464
PR #66072090833818511478784
PR #59582090833794218070016
PR #66352090803404556210176
PR #66462090803025537929216
PR #66472090794933425278976
PR #65832090753518062276608
PR #66402090745162878685184
PR #66252090743357851570176
PR #66292090725692609138688
PR #65192090717616074133504
PR #66122090717583954153472
PR #65192090683781416488960
PR #66122090678628760162304
PR #66412090669519650426880
PR #65842090659366838472704
PR #66292090657608682704896
PR #59582090649438694412288
PR #65902090647994339692544
PR #59582090619074294845440
PR #66382090612481398411264
PR #66302090598725549297664
PR #65902090587456947621888
PR #66382090580629858029568
PR #66122090577382321688576
PR #59582090576352783634432
PR #66072090575344808497152
PR #66302090566162734649344
PR #65902090555484766474240
PR #65842090554196767019008
PR #65842090553390097502208
PR #59582090549448319963136
PR #66312090548196454436864
PR #66332090523260918697984
PR #66322090522418434019328
PR #66312090514791213830144
PR #65842090514291462508544
PR #66292090501897923334144
PR #62782090489518846119936
PR #65752090484744067223552
PR #66252090479903731355648
PR #63632090463536332410880
PR #66072090448552185565184
PR #65922090394899235999744
PR #65752090373033880457216
PR #65832090364783722237952
PR #66122090355263239360512
PR #65192090346173553446912
PR #66072090344759288664064
PR #65832090334446552092672
PR #66162090329417594179584
PR #64922090283973790928896
PR #65572090266089802960896
PR #65992090253315500675072
PR #66122090248456953139200
PR #66142090214858837463040
PR #65882090212568353214464
PR #65992090212556621746176
PR #66072090206132864815104
PR #64922090198433062195200
PR #65842090192149080444928
PR #65722090182266104844288
PR #65992090179751976112128
PR #59582090166116277030912
PR #65842090159046848942080
PR #66082090119905717260288
PR #65992090119424534122496
PR #65922090105093117448192
PR #64262090100918157578240
PR #65192090085989094526976
PR #66002090085087973478400
PR #66022090055453999370240
PR #65992090035242617278464
PR #65992089998936298229760
PR #65832089982500192391168
PR #65832089907805891858432
PR #65582089876846182141952
PR #65932089864988096204800
PR #65572089843361052626944
PR #65582089839143172444160
PR #65882089829163719462912
PR #65932089824392346341376
PR #62322089820335682621440
PR #65582089788762895683584
PR #64922089786820278620160
PR #65762089786196182962176
PR #65882089759198756737024
PR #65752089753182908452864
PR #65842089749669939777536
PR #45662089740517767122944
PR #65272089722221688786944
PR #65842089714962824957952
PR #65852089708056525934592
PR #65642089691879867158528
PR #65532089679732839813120
PR #65642089674709242220544
PR #65622089651610870353920
PR #65532089604166161797120
PR #65662089600967690424320
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
alert6511.07%65 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsAug 17: 9 · Aug 18: 14 · Aug 19: 9 · Aug 20: 14 · Aug 21: 12 · Aug 22: 2 · Aug 23: 0 · Aug 24: 14none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-25T00:35:25Z
  Ended:       2026-08-25T00:56:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1843200-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd9551d19b6bblvufm3l3a421htavmi-a4k0z5e8x0e3t0f", pod="router-597d99bc98-tlmg9", severity="warning"
  Description: Pod ocm-arohcpci01-2sd9551d19b6bblvufm3l3a421htavmi-a4k0z5e8x0e3t0f/router-597d99bc98-tlmg9 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T23:47:43Z
  Ended:       2026-08-25T00:09:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4709760-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd8f93h71j1qrcah2q03eja41gojmif-i6w1a1c6f6m8f7l", pod="router-7f97cf9c5-phjp6", severity="warning"
  Description: Pod ocm-arohcpci01-2sd8f93h71j1qrcah2q03eja41gojmif-i6w1a1c6f6m8f7l/router-7f97cf9c5-phjp6 has been in a non-ready state for longer than 5 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T22:04:58Z
  Ended:       2026-08-24T22:16:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9409536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6tku01d4e5c0s0k2ocfdtf91cpr8l-d3q6r6q7q0m9f7w", pod="router-57846fcf5b-n5mlw", severity="warning"
  Description: Pod ocm-arohcpci01-2sd6tku01d4e5c0s0k2ocfdtf91cpr8l-d3q6r6q7q0m9f7w/router-57846fcf5b-n5mlw has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-24T22:04:58Z
  Ended:       2026-08-24T22:16:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9409536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sd6td57g5fetk5hnhg543eg0jfi9lrt-j5s2f3c0f5x7m8z", pod="router-7586ffccfd-4mvlg", severity="warning"
  Description: Pod ocm-arohcpci01-2sd6td57g5fetk5hnhg543eg0jfi9lrt-j5s2f3c0f5x7m8z/router-7586ffccfd-4mvlg has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire65
Affected runs (65)
Date (UTC)Associated PRProw job
PR #65542092034671041843200
PR #66352092020242174709760
PR #66602091994848889409536
PR #63642091989769276887040
PR #66412091984065988136960
PR #65832091965378614792192
PR #66482091962153304068096
PR #59582091939168878858240
PR #66572091928139092463616
PR #65842091920407983558656
PR #66292091899442289446912
PR #63642091898914398539776
PR #65752091896822531362816
PR #62322091892836139012096
PR #65932091039068275085312
PR #65572091023619198029824
PR #65802090930773002555392
PR #66312090908436404375552
PR #66482090882234021056512
PR #65932090879591731171328
PR #66522090868362149105664
PR #66502090866482933141504
PR #66482090836156110344192
PR #66122090807183972241408
PR #65832090753518062276608
PR #66412090669519650426880
PR #63632090657643948412928
PR #66292090657608682704896
PR #66122090577382321688576
PR #66312090576104745078784
PR #62322090572687599472640
PR #59582090549448319963136
PR #66212090514755771961344
PR #65842090514291462508544
PR #62782090489518846119936
PR #66172090419688172425216
PR #66122090389808017838080
PR #66162090329417594179584
PR #65992090313734160388096
PR #65842090249792243372032
PR #66122090248456953139200
PR #65552090233576732758016
PR #65842090221977028530176
PR #66082090191599463043072
PR #59582090166116277030912
PR #65992090146594451099648
PR #66112090141674171797504
PR #66072090085264507539456
PR #60702090078040460627968
PR #66062090076575402823680
PR #66002090054287706034176
PR #65882089829163719462912
PR #65622089810122246197248
PR #65882089759198756737024
PR #45662089740517767122944
PR #65272089725920976834560
PR #61572089723779814002688
PR #65832089713360550498304
PR #65862089708355189739520
PR #65852089708056525934592
PR #60702089685516600479744
PR #65532089679732839813120
PR #65732089665224855523328
PR #65622089651610870353920
PR #65662089560445248606208
alert [svc] KubeDaemonSetRolloutStuck fired
full failure pattern:
alert [svc] KubeDaemonSetRolloutStuck fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert498.35%49 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 16 · Aug 18: 15 · Aug 19: 12 · Aug 20: 10 · Aug 21: 11 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
alert fired 24 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T16:11:08Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:25:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-08-24T16:13:07Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-08-24T16:13:07Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:26:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:24:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-08-24T16:17:09Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-08-24T16:17:09Z
  Ended:       2026-08-24T16:23:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-08-24T16:17:09Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-08-24T16:17:09Z
  Ended:       2026-08-24T16:29:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
alert fired 24 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:53:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:53:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:53:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:53:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
alert fired 24 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-provider-azure", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-provider-azure has not finished or progressed for at least 15 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-npm", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-npm has not finished or progressed for at least 15 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azurefile-node", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azurefile-node has not finished or progressed for at least 15 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="kube-proxy", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/kube-proxy has not finished or progressed for at least 15 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="csi-azuredisk-node", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/csi-azuredisk-node has not finished or progressed for at least 15 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="cloud-node-manager", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/cloud-node-manager has not finished or progressed for at least 15 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:52:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:52:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet velero/node-agent has not finished or progressed for at least 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire49
Affected runs (49)
Date (UTC)Associated PRProw job
PR #59582091901058069565440
PR #66322090913249020088320
PR #66512090912033770835968
PR #65832090894675127832576
PR #65832090849031038177280
PR #65902090767289052827648
PR #65752090676319984553984
PR #65842090660118717796352
PR #63632090657643948412928
PR #66292090657608682704896
PR #65752090643626764800000
PR #66122090612430223708160
PR #65902090587456947621888
PR #65842090514291462508544
PR #65902090497381098328064
PR #65802090496257268125696
PR #66212090473213871001600
PR #65802090443849187987456
PR #66172090419688172425216
PR #65892090373215313465344
PR #65832090364783722237952
PR #65192090346173553446912
PR #66142090180258316685312
PR #66112090141674171797504
PR #65842090126361980047360
PR #66082090119905717260288
PR #64922090111171486027776
PR #66082090087196852752384
PR #65532090087137016811520
PR #60702090078040460627968
PR #65992090072497390096384
PR #65992090035242617278464
PR #65832090031884472946688
PR #66022090018285352390656
PR #65882089829163719462912
PR #65622089809279165927424
PR #65922089796885723222016
PR #63632089768451781431296
PR #65882089759198756737024
PR #58802089757851600818176
PR #65752089753182908452864
PR #65272089725920976834560
PR #61572089723779814002688
PR #65832089713360550498304
PR #65852089708056525934592
PR #64262089706907987087360
PR #58802089686738145382400
PR #60702089685516600479744
PR #65732089665224855523328
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
e2e457.67%45 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 33 · Aug 18: 43 · Aug 19: 30 · Aug 20: 18 · Aug 21: 17 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1INT, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000e8af48>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mb2sm98hvbm8" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9",
      "name": "76928be7-64f8-4a4c-85f6-cd6b790148b9",
      "status": "Failed",
      "startTime": "2026-08-24T15:14:43.826676557Z",
      "endTime": "2026-08-24T15:38:50.051171369Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tm25j05l3q2f8a1e83cjmn4lda8g still exists (deletion dispatched at 2026-08-24T15:14:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000e8af48>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mb2sm98hvbm8" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76928be7-64f8-4a4c-85f6-cd6b790148b9",
      "name": "76928be7-64f8-4a4c-85f6-cd6b790148b9",
      "status": "Failed",
      "startTime": "2026-08-24T15:14:43.826676557Z",
      "endTime": "2026-08-24T15:38:50.051171369Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tm25j05l3q2f8a1e83cjmn4lda8g still exists (deletion dispatched at 2026-08-24T15:14:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000efa360>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-hzr77tbdkrwt" to finish deleting: GET https://rp.j0088320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf6ae19-b7a5-40cd-806b-5002e0ccbaca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf6ae19-b7a5-40cd-806b-5002e0ccbaca",
      "name": "9bf6ae19-b7a5-40cd-806b-5002e0ccbaca",
      "status": "Failed",
      "startTime": "2026-08-21T22:23:11.170362844Z",
      "endTime": "2026-08-21T22:47:12.159444276Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7tt2c2j1a6r0ch04519gisqcoillq still exists (deletion dispatched at 2026-08-21T22:23:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000efa360>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-hzr77tbdkrwt" to finish deleting: GET https://rp.j0088320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf6ae19-b7a5-40cd-806b-5002e0ccbaca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf6ae19-b7a5-40cd-806b-5002e0ccbaca",
      "name": "9bf6ae19-b7a5-40cd-806b-5002e0ccbaca",
      "status": "Failed",
      "startTime": "2026-08-21T22:23:11.170362844Z",
      "endTime": "2026-08-21T22:47:12.159444276Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7tt2c2j1a6r0ch04519gisqcoillq still exists (deletion dispatched at 2026-08-21T22:23:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00182a330>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-p78v2jfc6wjs" to finish deleting: GET https://rp.j0835968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a8467ce1-61c9-4bf1-976b-a21d431ed9d8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a8467ce1-61c9-4bf1-976b-a21d431ed9d8",
      "name": "a8467ce1-61c9-4bf1-976b-a21d431ed9d8",
      "status": "Failed",
      "startTime": "2026-08-21T22:56:57.039348438Z",
      "endTime": "2026-08-21T23:21:07.00564298Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7rc59dodo08pmvlbds9o0qolurven still exists (deletion dispatched at 2026-08-21T22:56:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00182a330>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-p78v2jfc6wjs" to finish deleting: GET https://rp.j0835968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a8467ce1-61c9-4bf1-976b-a21d431ed9d8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a8467ce1-61c9-4bf1-976b-a21d431ed9d8",
      "name": "a8467ce1-61c9-4bf1-976b-a21d431ed9d8",
      "status": "Failed",
      "startTime": "2026-08-21T22:56:57.039348438Z",
      "endTime": "2026-08-21T23:21:07.00564298Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7rc59dodo08pmvlbds9o0qolurven still exists (deletion dispatched at 2026-08-21T22:56:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (44)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE can pause schedules to stop backup execution for an HCP cluster8
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.226
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.224
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.204
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.192
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (45)
Date (UTC)Associated PRProw job
PR #62322091892836139012096
PR #66322090913249020088320
PR #66512090912033770835968
PR #65832090894675127832576
PR #65902090767289052827648
PR #65752090676319984553984
PR #65842090660118717796352
PR #63632090657643948412928
PR #66292090657608682704896
PR #65752090643626764800000
PR #65902090587456947621888
PR #66382090580629858029568
PR #65842090514291462508544
PR #65802090496257268125696
PR #66212090473213871001600
PR #65802090443849187987456
PR #65892090373215313465344
PR #65832090364783722237952
PR #65192090346173553446912
PR #65842090249792243372032
PR #66112090141674171797504
PR #65842090126361980047360
PR #66082090119905717260288
PR #64922090111171486027776
PR #66082090087196852752384
PR #66072090085264507539456
PR #66012090079394612645888
PR #65992090072497390096384
PR #65992090035242617278464
PR #66022090018285352390656
PR #61322089850706507862016
PR #65882089829163719462912
PR #65622089809279165927424
PR #65922089796885723222016
PR #63632089768451781431296
PR #65882089759198756737024
PR #58802089757851600818176
PR #65752089753182908452864
PR #61572089723779814002688
PR #65832089713360550498304
PR #65852089708056525934592
PR #64262089706907987087360
PR #58802089686738145382400
PR #65732089665224855523328
PR #65622089651610870353920
alert [svc] KubeDaemonSetMisScheduled fired
full failure pattern:
alert [svc] KubeDaemonSetMisScheduled fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert447.50%44 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 15 · Aug 18: 15 · Aug 19: 9 · Aug 20: 9 · Aug 21: 10 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Firing 2:
  State:       Resolved
  Started:     2026-08-24T16:12:09Z
  Ended:       2026-08-24T16:24:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 3:
  State:       Resolved
  Started:     2026-08-24T16:13:10Z
  Ended:       2026-08-24T16:23:08Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 4:
  State:       Resolved
  Started:     2026-08-24T16:13:11Z
  Ended:       2026-08-24T16:23:08Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="velero", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:50:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="velero", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetMisScheduled does not fire44
Affected runs (44)
Date (UTC)Associated PRProw job
PR #59582091901058069565440
PR #66322090913249020088320
PR #66512090912033770835968
PR #65832090894675127832576
PR #65832090849031038177280
PR #65902090767289052827648
PR #65752090676319984553984
PR #65842090660118717796352
PR #63632090657643948412928
PR #66292090657608682704896
PR #66122090612430223708160
PR #65902090587456947621888
PR #65842090514291462508544
PR #65902090497381098328064
PR #65802090496257268125696
PR #66212090473213871001600
PR #65802090443849187987456
PR #65892090373215313465344
PR #65832090364783722237952
PR #65192090346173553446912
PR #66112090141674171797504
PR #65842090126361980047360
PR #66082090119905717260288
PR #64922090111171486027776
PR #66082090087196852752384
PR #65532090087137016811520
PR #60702090078040460627968
PR #65992090072497390096384
PR #65992090035242617278464
PR #65882089829163719462912
PR #65622089809279165927424
PR #63632089768451781431296
PR #65882089759198756737024
PR #58802089757851600818176
PR #65752089753182908452864
PR #65272089725920976834560
PR #61572089723779814002688
PR #65832089713360550498304
PR #65852089708056525934592
PR #64262089706907987087360
PR #58802089686738145382400
PR #60702089685516600479744
PR #65732089665224855523328
PR #65622089651610870353920
alert [svc] BackendControllerRetryHotLoop fired
full failure pattern:
alert [svc] BackendControllerRetryHotLoop fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
alert447.50%44 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsAug 17: 1 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 41 · Aug 22: 0 · Aug 23: 0 · Aug 24: 3none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-25T00:10:04Z
  Ended:       2026-08-25T00:34:04Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j8026752-svc", component="backend", name="observemanagedresourcegroup", severity="warning"
  Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T22:22:12Z
  Ended:       2026-08-24T22:46:11Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j2642688-svc", component="backend", name="observemanagedresourcegroup", severity="warning"
  Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T19:54:46Z
  Ended:       2026-08-24T20:17:49Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j4068096-svc", component="backend", name="observemanagedresourcegroup", severity="warning"
  Description: Backend controller workqueue observemanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire44
Affected runs (44)
Date (UTC)Associated PRProw job
PR #66482092031971118026752
PR #66482091995273302642688
PR #66482091962153304068096
PR #66072090894768476262400
PR #59582090894756736405504
PR #65832090894675127832576
PR #65932090878713473273856
PR #66122090877968120287232
PR #66512090871776832655360
PR #66502090866482933141504
PR #66122090849047588900864
PR #65832090849031038177280
PR #66322090837373561606144
PR #66312090837156841918464
PR #66472090836865526534144
PR #66482090836156110344192
PR #66072090833818511478784
PR #59582090833794218070016
PR #66122090807183972241408
PR #66312090804889495015424
PR #66352090803404556210176
PR #66472090794933425278976
PR #66432090790935381151744
PR #65752090784604393836544
PR #65192090780869554868224
PR #66312090773034322366464
PR #66462090768490683174912
PR #66432090757087574364160
PR #65832090753518062276608
PR #66252090725937179004928
PR #66292090725692609138688
PR #65192090717616074133504
PR #66122090717583954153472
PR #59582090711731759222784
PR #65832090693922316619776
PR #66072090688754837622784
PR #65192090683781416488960
PR #65902090677551943913472
PR #65842090660118717796352
PR #65842090659366838472704
PR #63632090657643948412928
PR #66292090657608682704896
PR #66382090644128227397632
PR #65752090643626764800000
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
alert396.64%39 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsAug 17: 10 · Aug 18: 16 · Aug 19: 6 · Aug 20: 8 · Aug 21: 5 · Aug 22: 3 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T18:15:21Z
  Ended:       2026-08-24T18:36:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-22T10:06:50Z
  Ended:       2026-08-22T10:14:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3442688-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.129:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3442688/providers/microsoft.monitor/accounts/hcps-j3442688", namespace="ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-hp77b", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 16 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-22T06:09:36Z
  Ended:       2026-08-22T06:19:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeeil0i6mt7oboc04nhglhna7qqum0-k2t7b3m6u6j2z9c", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeeil0i6mt7oboc04nhglhna7qqum0-k2t7b3m6u6j2z9c/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-22T06:10:37Z
  Ended:       2026-08-22T06:17:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeepf04jq44auu7rr8i1ok2dcjhtlu-h9f7b5h9h4k7x9r", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeepf04jq44auu7rr8i1ok2dcjhtlu-h9f7b5h9h4k7x9r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-08-22T06:10:37Z
  Ended:       2026-08-22T06:19:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeepf04jq44auu7rr8i1ok2dcjhtlu-h9f7b5h9h4k7x9r", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeepf04jq44auu7rr8i1ok2dcjhtlu-h9f7b5h9h4k7x9r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-08-22T06:10:37Z
  Ended:       2026-08-22T06:16:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeekbk6283ad3oksf8bsv67jh0l3j8-cilium-cl", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeekbk6283ad3oksf8bsv67jh0l3j8-cilium-cl/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-08-22T06:11:38Z
  Ended:       2026-08-22T06:31:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeee9sknhjmru28j861qqr85i8o22i-q0s4s8a3f6d6o5k", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeee9sknhjmru28j861qqr85i8o22i-q0s4s8a3f6d6o5k/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-08-22T06:11:38Z
  Ended:       2026-08-22T06:19:39Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeek9brcripht70j68qvmcbnvoclaj-d7k7d4n7p7p2q4v", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeek9brcripht70j68qvmcbnvoclaj-d7k7d4n7p7p2q4v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-08-22T06:11:38Z
  Ended:       2026-08-22T06:19:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeeir6uue5hi968m1tal3vd4g0p4tk-r4y1m1u1q4k5j4w", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeeir6uue5hi968m1tal3vd4g0p4tk-r4y1m1u1q4k5j4w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-08-22T06:11:38Z
  Ended:       2026-08-22T06:17:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeehpv4kuqpssgmdcbocp5oq1soheu-v6i5c0r2t6w5h2g", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeehpv4kuqpssgmdcbocp5oq1soheu-v6i5c0r2t6w5h2g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-08-22T06:11:38Z
  Ended:       2026-08-22T06:17:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeek9brcripht70j68qvmcbnvoclaj-d7k7d4n7p7p2q4v", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeek9brcripht70j68qvmcbnvoclaj-d7k7d4n7p7p2q4v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-08-22T06:11:38Z
  Ended:       2026-08-22T06:17:37Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeempoqgao8mpiu344ftttlggj568f-private-ingress", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeempoqgao8mpiu344ftttlggj568f-private-ingress/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-08-22T06:12:37Z
  Ended:       2026-08-22T06:19:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeehpv4kuqpssgmdcbocp5oq1soheu-v6i5c0r2t6w5h2g", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeehpv4kuqpssgmdcbocp5oq1soheu-v6i5c0r2t6w5h2g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-08-22T06:12:37Z
  Ended:       2026-08-22T06:19:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeempoqgao8mpiu344ftttlggj568f-private-ingress", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeempoqgao8mpiu344ftttlggj568f-private-ingress/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-08-22T06:12:37Z
  Ended:       2026-08-22T06:31:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeee9sknhjmru28j861qqr85i8o22i-q0s4s8a3f6d6o5k", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeee9sknhjmru28j861qqr85i8o22i-q0s4s8a3f6d6o5k/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-08-22T06:12:37Z
  Ended:       2026-08-22T06:19:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeenmev23bl18jkrt2lmlsssg9k4gb-j0m5o8b7c0z0v9w", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeenmev23bl18jkrt2lmlsssg9k4gb-j0m5o8b7c0z0v9w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-08-22T06:13:36Z
  Ended:       2026-08-22T06:21:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbefdpdepfvfi4jatmrhbejsk2d6fbr-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbefdpdepfvfi4jatmrhbejsk2d6fbr-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-08-22T06:13:36Z
  Ended:       2026-08-22T06:19:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/hcps-j8029824", namespace="ocm-arohcpci01-2sbeep8cvaflqvdh2ollk224hmbcdpga-v20260901", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sbeep8cvaflqvdh2ollk224hmbcdpga-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire39
Affected runs (39)
Date (UTC)Associated PRProw job
PR #65752091931812560375808
PR #66402091085161763442688
PR #65572091023619198029824
PR #66392090960143322714112
PR #66322090913249020088320
PR #66512090912033770835968
PR #65902090767289052827648
PR #65752090676319984553984
PR #66292090657608682704896
PR #62322090572687599472640
PR #65842090514291462508544
PR #65902090497381098328064
PR #65802090496257268125696
PR #65572090465920358027264
PR #65832090364783722237952
PR #65192090346173553446912
PR #66122090248456953139200
PR #66072090206132864815104
PR #66112090141674171797504
PR #66012090133175878553600
PR #65842090126361980047360
PR #60702090078040460627968
PR #65992090072497390096384
PR #61322089850706507862016
PR #65882089829163719462912
PR #65622089809279165927424
PR #58802089757851600818176
PR #65532089754858566455296
PR #59002089753382901256192
PR #65752089753182908452864
PR #65272089725920976834560
PR #61572089723779814002688
PR #65832089713360550498304
PR #65852089708056525934592
PR #64262089706907987087360
PR #58802089686738145382400
PR #60702089685516600479744
PR #65732089665224855523328
PR #60702089644114403397632
alert [svc] KubeNodeNotReady fired
full failure pattern:
alert [svc] KubeNodeNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert386.47%38 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsAug 17: 9 · Aug 18: 12 · Aug 19: 10 · Aug 20: 8 · Aug 21: 8 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T23:03:52Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="prometheus", node="aks-userswft1-18345013-vmss000002", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-userswft1-18345013-vmss000002 has been unready for more than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T23:04:56Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="prometheus", node="aks-userswft1-18345013-vmss000002", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-userswft1-18345013-vmss000002 has been unready for more than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T23:03:53Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="prometheus", node="aks-userswft1-16379006-vmss000000", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-userswft1-16379006-vmss000000 has been unready for more than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T23:03:53Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="prometheus", node="aks-userswft1-16379006-vmss000000", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-userswft1-16379006-vmss000000 has been unready for more than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:02:48Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j7832576-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.184:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7832576/providers/microsoft.monitor/accounts/services-j7832576", namespace="prometheus", node="aks-userswft1-31866601-vmss000003", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-rfcdk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-userswft1-31866601-vmss000003 has been unready for more than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:03:45Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j7832576-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.184:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7832576/providers/microsoft.monitor/accounts/services-j7832576", namespace="prometheus", node="aks-userswft1-31866601-vmss000003", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-rfcdk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-userswft1-31866601-vmss000003 has been unready for more than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeNodeNotReady does not fire38
Affected runs (38)
Date (UTC)Associated PRProw job
PR #66322090913249020088320
PR #66512090912033770835968
PR #65832090894675127832576
PR #65902090767289052827648
PR #65752090676319984553984
PR #63632090657643948412928
PR #66292090657608682704896
PR #65752090643626764800000
PR #65902090587456947621888
PR #65842090514291462508544
PR #65802090496257268125696
PR #66212090473213871001600
PR #65802090443849187987456
PR #66172090419688172425216
PR #65832090364783722237952
PR #65192090346173553446912
PR #66142090180258316685312
PR #66112090141674171797504
PR #65842090126361980047360
PR #66082090119905717260288
PR #64922090111171486027776
PR #60702090078040460627968
PR #65992090072497390096384
PR #65992090035242617278464
PR #65832090031884472946688
PR #66022090018285352390656
PR #65882089829163719462912
PR #65622089809279165927424
PR #63632089768451781431296
PR #58802089757851600818176
PR #65752089753182908452864
PR #61572089723779814002688
PR #65832089713360550498304
PR #65852089708056525934592
PR #64262089706907987087360
PR #58802089686738145382400
PR #60702089685516600479744
PR #65732089665224855523328
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
e2e335.62%33 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 2 · Aug 18: 18 · Aug 19: 2 · Aug 20: 3 · Aug 21: 0 · Aug 22: 4 · Aug 23: 0 · Aug 24: 31INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-h74snzg5tpbh/cluster-candidate-5-0-gp9ztx should provision
Unexpected error:
    <*fmt.wrapError | 0xc0014a05c0>: 
    failed to create HCP cluster cluster-candidate-5-0-gp9ztx: failed waiting for cluster="cluster-candidate-5-0-gp9ztx" in resourcegroup="rg-candidate-5-0-h74snzg5tpbh" to finish creating: GET https://rp.j1843200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617",
      "name": "de4739f4-eebd-4fe2-bf80-3bf6d57ff617",
      "status": "Failed",
      "startTime": "2026-08-25T00:23:47.993778746Z",
      "endTime": "2026-08-25T00:42:49.076566651Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-h74snzg5tpbh/cluster-candidate-5-0-gp9ztx should provision
Unexpected error:
    <*fmt.wrapError | 0xc0014a05c0>: 
    failed to create HCP cluster cluster-candidate-5-0-gp9ztx: failed waiting for cluster="cluster-candidate-5-0-gp9ztx" in resourcegroup="rg-candidate-5-0-h74snzg5tpbh" to finish creating: GET https://rp.j1843200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de4739f4-eebd-4fe2-bf80-3bf6d57ff617",
      "name": "de4739f4-eebd-4fe2-bf80-3bf6d57ff617",
      "status": "Failed",
      "startTime": "2026-08-25T00:23:47.993778746Z",
      "endTime": "2026-08-25T00:42:49.076566651Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-x45vwlgpgvzk/cluster-candidate-4-21-rztpw4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f3e6a0>: 
    failed to create HCP cluster cluster-candidate-4-21-rztpw4: failed waiting for cluster="cluster-candidate-4-21-rztpw4" in resourcegroup="rg-candidate-4-21-x45vwlgpgvzk" to finish creating: GET https://rp.j4709760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c",
      "name": "e986c420-0503-4186-9a01-bad1f92a005c",
      "status": "Failed",
      "startTime": "2026-08-24T23:37:07.202301694Z",
      "endTime": "2026-08-24T23:56:10.677598487Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-x45vwlgpgvzk/cluster-candidate-4-21-rztpw4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f3e6a0>: 
    failed to create HCP cluster cluster-candidate-4-21-rztpw4: failed waiting for cluster="cluster-candidate-4-21-rztpw4" in resourcegroup="rg-candidate-4-21-x45vwlgpgvzk" to finish creating: GET https://rp.j4709760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e986c420-0503-4186-9a01-bad1f92a005c",
      "name": "e986c420-0503-4186-9a01-bad1f92a005c",
      "status": "Failed",
      "startTime": "2026-08-24T23:37:07.202301694Z",
      "endTime": "2026-08-24T23:56:10.677598487Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d48840>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-mglhphvdtjhd" to finish creating: GET https://rp.j6339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb",
      "name": "1dbfdf90-d2f5-4efc-b68b-42b49d8220eb",
      "status": "Failed",
      "startTime": "2026-08-24T23:29:18.405302537Z",
      "endTime": "2026-08-24T23:48:26.458954406Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d48840>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-mglhphvdtjhd" to finish creating: GET https://rp.j6339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1dbfdf90-d2f5-4efc-b68b-42b49d8220eb",
      "name": "1dbfdf90-d2f5-4efc-b68b-42b49d8220eb",
      "status": "Failed",
      "startTime": "2026-08-24T23:29:18.405302537Z",
      "endTime": "2026-08-24T23:48:26.458954406Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (20)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.225
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE can pause schedules to stop backup execution for an HCP cluster5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.04
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM1
Affected runs (33)
Date (UTC)Associated PRProw job
PR #65542092034671041843200
PR #66352092020242174709760
PR #66632092019318316339200
PR #66582092016236664721408
PR #66602091994848889409536
PR #63642091989769276887040
PR #66412091984065988136960
PR #65832091965378614792192
PR #66382091963250454302720
PR #66482091962153304068096
PR #59582091939168878858240
PR #66292091932257060130816
PR #65912091928685790629888
PR #66572091928139092463616
PR #66292091899442289446912
PR #63642091898914398539776
PR #66582091895460682797056
PR #64292091879954055696384
PR #64332091879927736438784
PR #60702091855258673221632
PR #65552091237855425728512
PR #65802091230843602210816
PR #66122090577382321688576
PR #65842090554196767019008
PR #65552090233576732758016
PR #59582090166116277030912
PR #65842090093258611888128
PR #65812089746333521940480
PR #65642089725877364461568
PR #61572089723779814002688
PR #65642089690861355929600
PR #60702089685516600479744
PR #65532089679732839813120
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
e2e315.28%31 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 23 · Aug 18: 17 · Aug 19: 12 · Aug 20: 13 · Aug 21: 10 · Aug 22: 1 · Aug 23: 0 · Aug 24: 0INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001bdc3d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-k276rwlx6nvv" to finish deleting: GET https://rp.j4608896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95be590e-be2d-4405-b000-e47e729c722a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95be590e-be2d-4405-b000-e47e729c722a",
      "name": "95be590e-be2d-4405-b000-e47e729c722a",
      "status": "Failed",
      "startTime": "2026-08-22T01:29:12.661843478Z",
      "endTime": "2026-08-22T01:53:12.268348605Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sbairsosicqjiq127hgv6ukup2i9asv still exists (deletion dispatched at 2026-08-22T01:29:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001bdc3d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-k276rwlx6nvv" to finish deleting: GET https://rp.j4608896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95be590e-be2d-4405-b000-e47e729c722a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95be590e-be2d-4405-b000-e47e729c722a",
      "name": "95be590e-be2d-4405-b000-e47e729c722a",
      "status": "Failed",
      "startTime": "2026-08-22T01:29:12.661843478Z",
      "endTime": "2026-08-22T01:53:12.268348605Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sbairsosicqjiq127hgv6ukup2i9asv still exists (deletion dispatched at 2026-08-22T01:29:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc002b983a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-8fhdctfdz8rw" to finish deleting: GET https://rp.j0088320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d2b6d51-1179-46e6-9785-09bffda18aba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d2b6d51-1179-46e6-9785-09bffda18aba",
      "name": "1d2b6d51-1179-46e6-9785-09bffda18aba",
      "status": "Failed",
      "startTime": "2026-08-21T22:37:19.756260961Z",
      "endTime": "2026-08-21T23:01:22.169782395Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7u2scpkh7308u4g26d4adn4v6f2fq still exists (deletion dispatched at 2026-08-21T22:37:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc002b983a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-8fhdctfdz8rw" to finish deleting: GET https://rp.j0088320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d2b6d51-1179-46e6-9785-09bffda18aba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1d2b6d51-1179-46e6-9785-09bffda18aba",
      "name": "1d2b6d51-1179-46e6-9785-09bffda18aba",
      "status": "Failed",
      "startTime": "2026-08-21T22:37:19.756260961Z",
      "endTime": "2026-08-21T23:01:22.169782395Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7u2scpkh7308u4g26d4adn4v6f2fq still exists (deletion dispatched at 2026-08-21T22:37:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00160c810>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7ckkp246v7kn" to finish deleting: GET https://rp.j0088320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/daa2ac24-e65d-4ec6-b703-b65cb92de73e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/daa2ac24-e65d-4ec6-b703-b65cb92de73e",
      "name": "daa2ac24-e65d-4ec6-b703-b65cb92de73e",
      "status": "Failed",
      "startTime": "2026-08-21T22:26:31.187721464Z",
      "endTime": "2026-08-21T22:50:32.176977266Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7tr4tpdqmibta8tjhvqb9049in5ek still exists (deletion dispatched at 2026-08-21T22:26:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00160c810>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7ckkp246v7kn" to finish deleting: GET https://rp.j0088320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/daa2ac24-e65d-4ec6-b703-b65cb92de73e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/daa2ac24-e65d-4ec6-b703-b65cb92de73e",
      "name": "daa2ac24-e65d-4ec6-b703-b65cb92de73e",
      "status": "Failed",
      "startTime": "2026-08-21T22:26:31.187721464Z",
      "endTime": "2026-08-21T22:50:32.176977266Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb7tr4tpdqmibta8tjhvqb9049in5ek still exists (deletion dispatched at 2026-08-21T22:26:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (25)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling7
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (31)
Date (UTC)Associated PRProw job
PR #65932090955578854608896
PR #66322090913249020088320
PR #66512090912033770835968
PR #65832090849031038177280
PR #65902090767289052827648
PR #63632090657643948412928
PR #66292090657608682704896
PR #66122090612430223708160
PR #65842090514291462508544
PR #65902090497381098328064
PR #65802090496257268125696
PR #66212090473213871001600
PR #65832090364783722237952
PR #65192090346173553446912
PR #66112090141674171797504
PR #64922090111171486027776
PR #65532090087137016811520
PR #66012090079394612645888
PR #65992090035242617278464
PR #66022090018285352390656
PR #61322089850706507862016
PR #65942089843707992870912
PR #65882089829163719462912
PR #65622089809279165927424
PR #58802089757851600818176
PR #65752089753182908452864
PR #61572089723779814002688
PR #65852089708056525934592
PR #64262089706907987087360
PR #58802089686738145382400
PR #65732089665224855523328
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
alert315.28%31 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsAug 17: 9 · Aug 18: 10 · Aug 19: 5 · Aug 20: 6 · Aug 21: 5 · Aug 22: 3 · Aug 23: 0 · Aug 24: 2none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T18:01:25Z
  Ended:       2026-08-24T18:34:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-24T18:01:25Z
  Ended:       2026-08-24T18:34:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T17:55:40Z
  Ended:       2026-08-24T18:14:40Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0629888-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.209:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0629888/providers/microsoft.monitor/accounts/hcps-j0629888", namespace="ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-tktc6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-24T17:55:40Z
  Ended:       2026-08-24T18:15:41Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0629888-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.209:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0629888/providers/microsoft.monitor/accounts/hcps-j0629888", namespace="ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-tktc6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sd354mcvucng8mm6igmcq4k43caf9jb-s0d4l4d8t6j0p0t/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-22T09:51:47Z
  Ended:       2026-08-22T10:13:48Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3442688-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.129:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3442688/providers/microsoft.monitor/accounts/hcps-j3442688", namespace="ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-hp77b", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-22T09:52:47Z
  Ended:       2026-08-22T10:09:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3442688-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.129:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3442688/providers/microsoft.monitor/accounts/hcps-j3442688", namespace="ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-hp77b", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire31
Affected runs (31)
Date (UTC)Associated PRProw job
PR #65752091931812560375808
PR #65912091928685790629888
PR #66402091085161763442688
PR #65572091023619198029824
PR #66392090960143322714112
PR #65802090930773002555392
PR #66512090912033770835968
PR #66122090807183972241408
PR #66412090669519650426880
PR #66292090657608682704896
PR #62322090572687599472640
PR #65842090514291462508544
PR #65902090497381098328064
PR #65572090465920358027264
PR #65842090405439429677056
PR #66122090248456953139200
PR #66082090191599463043072
PR #66112090141674171797504
PR #66012090133175878553600
PR #66072090085264507539456
PR #60702090078040460627968
PR #65882089829163719462912
PR #65882089759198756737024
PR #65532089754858566455296
PR #59002089753382901256192
PR #65272089725920976834560
PR #61572089723779814002688
PR #65832089713360550498304
PR #65852089708056525934592
PR #64262089706907987087360
PR #60702089685516600479744
alert [svc] KubeNodeUnreachable fired
full failure pattern:
alert [svc] KubeNodeUnreachable fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert284.77%28 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 12 · Aug 18: 7 · Aug 19: 4 · Aug 20: 7 · Aug 21: 9 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T16:12:59Z
  Ended:       2026-08-24T16:22:56Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="prometheus", node="aks-userswft2-20578738-vmss000002", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft2-20578738-vmss000002 is unreachable and some workloads may be rescheduled.
Firing 2:
  State:       Resolved
  Started:     2026-08-24T16:12:59Z
  Ended:       2026-08-24T16:23:55Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j9565440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9565440/providers/microsoft.monitor/accounts/services-j9565440", namespace="prometheus", node="aks-userswft2-20578738-vmss000002", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-wgnw8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft2-20578738-vmss000002 is unreachable and some workloads may be rescheduled.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:52Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="prometheus", node="aks-userswft1-18345013-vmss000002", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-18345013-vmss000002 is unreachable and some workloads may be rescheduled.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:49:52Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j0088320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.93:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0088320/providers/microsoft.monitor/accounts/services-j0088320", namespace="prometheus", node="aks-userswft1-18345013-vmss000002", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-vxkdd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-18345013-vmss000002 is unreachable and some workloads may be rescheduled.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="prometheus", node="aks-userswft1-16379006-vmss000000", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-16379006-vmss000000 is unreachable and some workloads may be rescheduled.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T22:48:54Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j0835968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.64.147:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0835968/providers/microsoft.monitor/accounts/services-j0835968", namespace="prometheus", node="aks-userswft1-16379006-vmss000000", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-cxxjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-16379006-vmss000000 is unreachable and some workloads may be rescheduled.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeNodeUnreachable does not fire28
Affected runs (28)
Date (UTC)Associated PRProw job
PR #59582091901058069565440
PR #66322090913249020088320
PR #66512090912033770835968
PR #65832090894675127832576
PR #65832090849031038177280
PR #65902090767289052827648
PR #65752090676319984553984
PR #63632090657643948412928
PR #66292090657608682704896
PR #66122090612430223708160
PR #65902090587456947621888
PR #65842090514291462508544
PR #65802090496257268125696
PR #65802090443849187987456
PR #65892090373215313465344
PR #65832090364783722237952
PR #65192090346173553446912
PR #66112090141674171797504
PR #60702090078040460627968
PR #65992090035242617278464
PR #66022090018285352390656
PR #65882089829163719462912
PR #65622089809279165927424
PR #65882089759198756737024
PR #58802089757851600818176
PR #61572089723779814002688
PR #60702089685516600479744
PR #65622089651610870353920
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
e2e274.60%27 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsAug 17: 29 · Aug 18: 26 · Aug 19: 14 · Aug 20: 33 · Aug 21: 40 · Aug 22: 14 · Aug 23: 0 · Aug 24: 1INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-nzmkwtbzb8xk/cluster-candidate-4-21-k47dlw should provision
Unexpected error:
    <*fmt.wrapError | 0xc00011d300>: 
    failed to create HCP cluster cluster-candidate-4-21-k47dlw: failed waiting for cluster="cluster-candidate-4-21-k47dlw" in resourcegroup="rg-candidate-4-21-nzmkwtbzb8xk" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd",
      "name": "e224492a-2630-4a37-b5e0-7ef00f6cd9dd",
      "status": "Failed",
      "startTime": "2026-08-24T17:34:09.560283017Z",
      "endTime": "2026-08-24T17:53:13.775593008Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-nzmkwtbzb8xk/cluster-candidate-4-21-k47dlw should provision
Unexpected error:
    <*fmt.wrapError | 0xc00011d300>: 
    failed to create HCP cluster cluster-candidate-4-21-k47dlw: failed waiting for cluster="cluster-candidate-4-21-k47dlw" in resourcegroup="rg-candidate-4-21-nzmkwtbzb8xk" to finish creating: GET https://rp.j0629888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e224492a-2630-4a37-b5e0-7ef00f6cd9dd",
      "name": "e224492a-2630-4a37-b5e0-7ef00f6cd9dd",
      "status": "Failed",
      "startTime": "2026-08-24T17:34:09.560283017Z",
      "endTime": "2026-08-24T17:53:13.775593008Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001002c80>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-fvxb2h745mmv" to finish creating: GET https://rp.j3442688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/927301b0-80ab-45c1-a577-7613e32d2b80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/927301b0-80ab-45c1-a577-7613e32d2b80",
      "name": "927301b0-80ab-45c1-a577-7613e32d2b80",
      "status": "Failed",
      "startTime": "2026-08-22T09:32:34.244455805Z",
      "endTime": "2026-08-22T09:51:43.68061461Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001002c80>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-fvxb2h745mmv" to finish creating: GET https://rp.j3442688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/927301b0-80ab-45c1-a577-7613e32d2b80
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/927301b0-80ab-45c1-a577-7613e32d2b80",
      "name": "927301b0-80ab-45c1-a577-7613e32d2b80",
      "status": "Failed",
      "startTime": "2026-08-22T09:32:34.244455805Z",
      "endTime": "2026-08-22T09:51:43.68061461Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0009fd080>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-f7hhtzrszbl9" to finish creating: GET https://rp.j8029824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/edb9f485-31ea-4549-83fe-b228b1e92d7f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/edb9f485-31ea-4549-83fe-b228b1e92d7f",
      "name": "edb9f485-31ea-4549-83fe-b228b1e92d7f",
      "status": "Failed",
      "startTime": "2026-08-22T05:37:57.735593463Z",
      "endTime": "2026-08-22T05:57:05.414598604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0009fd080>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-f7hhtzrszbl9" to finish creating: GET https://rp.j8029824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/edb9f485-31ea-4549-83fe-b228b1e92d7f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/edb9f485-31ea-4549-83fe-b228b1e92d7f",
      "name": "edb9f485-31ea-4549-83fe-b228b1e92d7f",
      "status": "Failed",
      "startTime": "2026-08-22T05:37:57.735593463Z",
      "endTime": "2026-08-22T05:57:05.414598604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (50)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.217
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully7
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal7
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.05
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.225
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE can pause schedules to stop backup execution for an HCP cluster5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.224
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and successfully scale nodepool replicas1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a fully private cluster with private KAS, private ingress, and private KeyVault and verify all are operational1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelHCP creates on-demand backup after KMS key rotation completes1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (27)
Date (UTC)Associated PRProw job
PR #65912091928685790629888
PR #66402091085161763442688
PR #65572091023619198029824
PR #66392090960143322714112
PR #65542090958470403592192
PR #65802090930773002555392
PR #66122090807183972241408
PR #66412090669519650426880
PR #66292090657608682704896
PR #62322090572687599472640
PR #65842090514291462508544
PR #65902090497381098328064
PR #65572090465920358027264
PR #65842090405439429677056
PR #66122090248456953139200
PR #66082090191599463043072
PR #66012090133175878553600
PR #66072090085264507539456
PR #60702090078040460627968
PR #65882089829163719462912
PR #65882089759198756737024
PR #65532089754858566455296
PR #59002089753382901256192
PR #65272089725920976834560
PR #65832089713360550498304
PR #65852089708056525934592
PR #60702089685516600479744
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
e2e152.56%15 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsAug 17: 1 · Aug 18: 2 · Aug 19: 1 · Aug 20: 7 · Aug 21: 2 · Aug 22: 1 · Aug 23: 0 · Aug 24: 2INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-20 with version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc00031f900>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-tczrqg" in resourcegroup="rg-np-version-upgrade-tczrqg-fhlnbg67qjlz" to finish creating: GET https://rp.j7254272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377",
      "name": "c2d7b313-a7f0-40de-93b0-918f61dd2377",
      "status": "Failed",
      "startTime": "2026-08-24T20:58:35.24134299Z",
      "endTime": "2026-08-24T21:17:39.53487523Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-20 with version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc00031f900>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-tczrqg" in resourcegroup="rg-np-version-upgrade-tczrqg-fhlnbg67qjlz" to finish creating: GET https://rp.j7254272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2d7b313-a7f0-40de-93b0-918f61dd2377",
      "name": "c2d7b313-a7f0-40de-93b0-918f61dd2377",
      "status": "Failed",
      "startTime": "2026-08-24T20:58:35.24134299Z",
      "endTime": "2026-08-24T21:17:39.53487523Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:599]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000e96960>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-q2tthc" in resourcegroup="rg-np-downgrade-q2tthc-z7rbl7r2jwws" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5",
      "name": "47831462-3285-4a15-8722-8601b5708cf5",
      "status": "Failed",
      "startTime": "2026-08-24T15:29:26.286804415Z",
      "endTime": "2026-08-24T15:48:34.685207308Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:599]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000e96960>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-q2tthc" in resourcegroup="rg-np-downgrade-q2tthc-z7rbl7r2jwws" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/47831462-3285-4a15-8722-8601b5708cf5",
      "name": "47831462-3285-4a15-8722-8601b5708cf5",
      "status": "Failed",
      "startTime": "2026-08-24T15:29:26.286804415Z",
      "endTime": "2026-08-24T15:48:34.685207308Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:192]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0020dd2d8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-m6p9wwz6cdqp" to finish creating: GET https://rp.j4608896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bace4bf7-d284-4ee6-a7e4-bdbf462cf64f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bace4bf7-d284-4ee6-a7e4-bdbf462cf64f",
      "name": "bace4bf7-d284-4ee6-a7e4-bdbf462cf64f",
      "status": "Failed",
      "startTime": "2026-08-22T01:31:06.356215706Z",
      "endTime": "2026-08-22T01:50:12.095890348Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:192]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0020dd2d8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-m6p9wwz6cdqp" to finish creating: GET https://rp.j4608896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bace4bf7-d284-4ee6-a7e4-bdbf462cf64f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bace4bf7-d284-4ee6-a7e4-bdbf462cf64f",
      "name": "bace4bf7-d284-4ee6-a7e4-bdbf462cf64f",
      "status": "Failed",
      "startTime": "2026-08-22T01:31:06.356215706Z",
      "endTime": "2026-08-22T01:50:12.095890348Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (11)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (15)
Date (UTC)Associated PRProw job
PR #66582091977503907254272
PR #65752091896822531362816
PR #65932090955578854608896
PR #66352090803404556210176
PR #66122090612430223708160
PR #59582090576352783634432
PR #65842090514291462508544
PR #66292090501897923334144
PR #65902090497381098328064
PR #65752090484744067223552
PR #65752090483907651702784
PR #65892090373215313465344
PR #61572089981796769861632
PR #62322089820335682621440
PR #64922089679544620421120
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
e2e132.21%13 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 2 · Aug 18: 3 · Aug 19: 2 · Aug 20: 0 · Aug 21: 27 · Aug 22: 5 · Aug 23: 0 · Aug 24: 4PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000da71c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wqnnn79bk4qc" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673",
      "name": "e635655f-c9ac-44e1-8232-3550a4c20673",
      "status": "Failed",
      "startTime": "2026-08-24T21:28:57.457434277Z",
      "endTime": "2026-08-24T21:48:06.423004565Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000da71c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-wqnnn79bk4qc" to finish creating: GET https://rp.j6887040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e635655f-c9ac-44e1-8232-3550a4c20673",
      "name": "e635655f-c9ac-44e1-8232-3550a4c20673",
      "status": "Failed",
      "startTime": "2026-08-24T21:28:57.457434277Z",
      "endTime": "2026-08-24T21:48:06.423004565Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000fa81a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jhvt8ptm7fw8" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2",
      "name": "12485de2-db77-439f-89df-5bacb5d218b2",
      "status": "Failed",
      "startTime": "2026-08-24T15:41:38.67903036Z",
      "endTime": "2026-08-24T16:00:39.109568382Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000fa81a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jhvt8ptm7fw8" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/12485de2-db77-439f-89df-5bacb5d218b2",
      "name": "12485de2-db77-439f-89df-5bacb5d218b2",
      "status": "Failed",
      "startTime": "2026-08-24T15:41:38.67903036Z",
      "endTime": "2026-08-24T16:00:39.109568382Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000e86920>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-7vpnvrbqnldf" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83",
      "name": "73cc6203-30af-4c53-9024-398221683c83",
      "status": "Failed",
      "startTime": "2026-08-24T15:17:58.555796542Z",
      "endTime": "2026-08-24T15:37:08.01693151Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000e86920>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-7vpnvrbqnldf" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/73cc6203-30af-4c53-9024-398221683c83",
      "name": "73cc6203-30af-4c53-9024-398221683c83",
      "status": "Failed",
      "startTime": "2026-08-24T15:17:58.555796542Z",
      "endTime": "2026-08-24T15:37:08.01693151Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (32)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE can pause schedules to stop backup execution for an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (13)
Date (UTC)Associated PRProw job
PR #63642091989769276887040
PR #63642091898914398539776
PR #65752091896822531362816
PR #66582091895460682797056
PR #65572090958673797976064
PR #65542090958470403592192
PR #65932090954745601593344
PR #66172090617602316439552
PR #65962090162950290542592
PR #65992090146594451099648
PR #45662089740517767122944
PR #61572089723779814002688
PR #65532089679732839813120
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
e2e132.21%13 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 0 · Aug 18: 3 · Aug 19: 2 · Aug 20: 1 · Aug 21: 4 · Aug 22: 2 · Aug 23: 0 · Aug 24: 1STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:208]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc00396d0e0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-4984p9m7f6lc" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4984p9m7f6lc, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:208]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc00396d0e0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-4984p9m7f6lc" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-4984p9m7f6lc, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:245]: failed to scale down node pool "np-scale" from 2 to 1 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc002c86f00>: 
    failed waiting for nodepool="np-scale" in cluster="priv-np-scale" resourcegroup="priv-np-scale-qtxpc74p5cw9" to finish updating, caused by: timeout '60.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-scale in cluster priv-np-scale in resource group priv-np-scale-qtxpc74p5cw9, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:245]: failed to scale down node pool "np-scale" from 2 to 1 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc002c86f00>: 
    failed waiting for nodepool="np-scale" in cluster="priv-np-scale" resourcegroup="priv-np-scale-qtxpc74p5cw9" to finish updating, caused by: timeout '60.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-scale in cluster priv-np-scale in resource group priv-np-scale-qtxpc74p5cw9, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:245]: failed to scale down node pool "np-scale" from 2 to 1 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d67b0>: 
    failed waiting for nodepool="np-scale" in cluster="priv-np-scale" resourcegroup="priv-np-scale-fcxcqdz2d2x2" to finish updating, caused by: timeout '40.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-scale in cluster priv-np-scale in resource group priv-np-scale-fcxcqdz2d2x2, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:245]: failed to scale down node pool "np-scale" from 2 to 1 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d67b0>: 
    failed waiting for nodepool="np-scale" in cluster="priv-np-scale" resourcegroup="priv-np-scale-fcxcqdz2d2x2" to finish updating, caused by: timeout '40.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-scale in cluster priv-np-scale in resource group priv-np-scale-fcxcqdz2d2x2, error: context deadline exceeded
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling9
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and successfully scale nodepool replicas3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
Affected runs (13)
Date (UTC)Associated PRProw job
PR #60702091855258673221632
PR #65802091176636450869248
PR #65802090989679758282752
PR #65802090884661910704128
PR #59582090833794218070016
PR #60702090817534927310848
PR #60702090766839033368576
PR #66122090355263239360512
PR #65722090181452632166400
PR #60702090078040460627968
PR #65892089793767912509440
PR #60702089685516600479744
PR #60702089644114403397632
ERROR CODE: DeploymentNotFound
full failure pattern:
ERROR CODE: DeploymentNotFound
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
provision111.87%11 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsAug 17: 5 · Aug 18: 3 · Aug 19: 1 · Aug 20: 1 · Aug 21: 4 · Aug 22: 0 · Aug 23: 0 · Aug 24: 2none
Full failure examples (3)
time=2026-08-24T18:57:37.594Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 description="Step output\n  Kind: ARM\n  Template: ./../dev-infrastructure/templates/output-mgmt.bicep\n  Parameters: ./../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam"
time=2026-08-24T18:57:37.979Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1
time=2026-08-24T18:57:38.350Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 deployment=cfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8797056-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837
time=2026-08-24T18:57:38.400Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8797056-mgmt-1/providers/Microsoft.Resources/deployments/cfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837/operationStatuses/08584140090272578417\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'cfd650f4557893b92396842785417ac4f900952cc40f35ef046b64198c66a837' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-08-24T17:20:28.034Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 description="Step output\n  Kind: ARM\n  Template: ../../dev-infrastructure/templates/output-mgmt.bicep\n  Parameters: ../../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam"
time=2026-08-24T17:20:28.426Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1
time=2026-08-24T17:20:28.817Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 deployment=3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2076032-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23
time=2026-08-24T17:20:28.866Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2076032-mgmt-1/providers/Microsoft.Resources/deployments/3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23/operationStatuses/08584140148567905734\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment '3472b226dd1bf3f5c2c2fbd8f62937b0fe2f97cb4ca8a63168999f9ba0945d23' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-08-21T16:46:53.918Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 description="Step output\n  Kind: ARM\n  Template: ./../dev-infrastructure/templates/output-mgmt.bicep\n  Parameters: ./../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam"
time=2026-08-21T16:46:54.412Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1
time=2026-08-21T16:46:54.927Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 deployment=d8863692838cbe3c0e6e8e47c2c166e7f2539a854071effa428b1dfa89818a4d portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0032384-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd8863692838cbe3c0e6e8e47c2c166e7f2539a854071effa428b1dfa89818a4d
time=2026-08-21T16:46:55.024Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0032384-mgmt-1/providers/Microsoft.Resources/deployments/d8863692838cbe3c0e6e8e47c2c166e7f2539a854071effa428b1dfa89818a4d/operationStatuses/08584142760707149990\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 Not Found\nERROR CODE: DeploymentNotFound\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentNotFound\",\n    \"message\": \"Deployment 'd8863692838cbe3c0e6e8e47c2c166e7f2539a854071effa428b1dfa89818a4d' could not be found.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Fleet.Registration/management/output (stamp=1)6
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Velero/management/output (stamp=1)5
Affected runs (11)
Date (UTC)Associated PRProw job
PR #65042091954836298797056
PR #45662091928527292076032
PR #66502090832700310032384
PR #65902090730615539240960
PR #65902090717629881782272
PR #66312090618928278540288
PR #60702090541078221426688
PR #65152090152370880122880
PR #65272089795500311056384
PR #65902089787112214761472
PR #65732089750086291558400
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.R...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
e2e101.70%10 of 587 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsAug 17: 0 · Aug 18: 3 · Aug 19: 2 · Aug 20: 1 · Aug 21: 1 · Aug 22: 1 · Aug 23: 0 · Aug 24: 4PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc00123e240>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2xcnpkr48kxk" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1",
      "name": "3185d8c9-e246-4082-b170-898da28192d1",
      "status": "Failed",
      "startTime": "2026-08-24T17:28:32.888311428Z",
      "endTime": "2026-08-24T17:47:34.534131157Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc00123e240>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2xcnpkr48kxk" to finish creating: GET https://rp.j2463616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3185d8c9-e246-4082-b170-898da28192d1",
      "name": "3185d8c9-e246-4082-b170-898da28192d1",
      "status": "Failed",
      "startTime": "2026-08-24T17:28:32.888311428Z",
      "endTime": "2026-08-24T17:47:34.534131157Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000dd0320>: 
    failed to create HCP cluster np-dg-minor-4tds4q: failed waiting for cluster="np-dg-minor-4tds4q" in resourcegroup="rg-np-dg-minor-4tds4q-6qtw9l2fwcdn" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5",
      "name": "162a10b5-886b-40a8-a95b-30fc38ce9bc5",
      "status": "Failed",
      "startTime": "2026-08-24T15:41:41.593852701Z",
      "endTime": "2026-08-24T16:00:49.106385261Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000dd0320>: 
    failed to create HCP cluster np-dg-minor-4tds4q: failed waiting for cluster="np-dg-minor-4tds4q" in resourcegroup="rg-np-dg-minor-4tds4q-6qtw9l2fwcdn" to finish creating: GET https://rp.j8539776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/162a10b5-886b-40a8-a95b-30fc38ce9bc5",
      "name": "162a10b5-886b-40a8-a95b-30fc38ce9bc5",
      "status": "Failed",
      "startTime": "2026-08-24T15:41:41.593852701Z",
      "endTime": "2026-08-24T16:00:49.106385261Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000e96560>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-fn2m5nmfttf7" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a",
      "name": "eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a",
      "status": "Failed",
      "startTime": "2026-08-24T15:17:08.176674236Z",
      "endTime": "2026-08-24T15:36:17.96458443Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000e96560>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-fn2m5nmfttf7" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a",
      "name": "eb7b7e77-c07f-48f4-a1bf-2f48bcb6434a",
      "status": "Failed",
      "startTime": "2026-08-24T15:17:08.176674236Z",
      "endTime": "2026-08-24T15:36:17.96458443Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
Affected runs (10)
Date (UTC)Associated PRProw job
PR #66572091928139092463616
PR #63642091898914398539776
PR #65752091896822531362816
PR #66582091895460682797056
PR #65932090954745601593344
PR #65752090643626764800000
PR #65842090314560790597632
PR #65992090146594451099648
PR #65272089725920976834560
PR #61572089723779814002688
UpdateHCPCluster20260630 timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster20260630 timed out after <minutes> minutes; context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
e2e101.70%10 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsAug 17: 0 · Aug 18: 3 · Aug 19: 3 · Aug 20: 1 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc00132c390>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cflktt657pch" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-cflktt657pch, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc00132c390>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cflktt657pch" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-cflktt657pch, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc00154c060>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hhjbvcghxvgq" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-hhjbvcghxvgq, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc00154c060>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hhjbvcghxvgq" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-hhjbvcghxvgq, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc000f7e900>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ntm46dnmmclw" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-ntm46dnmmclw, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:172]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc000f7e900>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ntm46dnmmclw" to finish updating, caused by: timeout '30.000000' minutes exceeded during UpdateHCPCluster20260630 for cluster kms-key-rotate-422 in resource group kms-key-rotate-ntm46dnmmclw, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.2210
Affected runs (10)
Date (UTC)Associated PRProw job
PR #60702091855258673221632
PR #60702090817534927310848
PR #60702090766839033368576
PR #65842090514291462508544
PR #65842090126361980047360
PR #66012090079394612645888
PR #60702090078040460627968
PR #60702089685516600479744
PR #65622089651610870353920
PR #60702089644114403397632
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
e2e91.53%9 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsAug 17: 3 · Aug 18: 1 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 0 · Aug 24: 6PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation9
Affected runs (9)
Date (UTC)Associated PRProw job
PR #65832091985911028912128
PR #65752091931812560375808
PR #65912091928685790629888
PR #66572091928139092463616
PR #59582091901058069565440
PR #66292091899442289446912
PR #65572091023619198029824
PR #60702090078040460627968
PR #65272089725920976834560
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert91.53%9 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsAug 17: 0 · Aug 18: 2 · Aug 19: 2 · Aug 20: 2 · Aug 21: 0 · Aug 22: 2 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T18:02:20Z
  Ended:       2026-08-24T18:38:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4cebd923-31c4-4c0a-b568-63bf1a281b9a"
  Description: Pod ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-08-24T18:03:20Z
  Ended:       2026-08-24T18:38:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/hcps-j0375808", namespace="ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4cebd923-31c4-4c0a-b568-63bf1a281b9a"
  Description: Pod ocm-arohcpci01-2sd37mkptl2qprba4ejkvlbldgl77u9k-r3i6h6d2n5v5b4i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-22T09:53:49Z
  Ended:       2026-08-22T10:12:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3442688-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.129:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3442688/providers/microsoft.monitor/accounts/hcps-j3442688", namespace="ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="73b11fbd-18cc-490d-bc49-f492731b9079"
  Description: Pod ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-08-22T09:54:46Z
  Ended:       2026-08-22T10:12:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3442688-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.129:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3442688/providers/microsoft.monitor/accounts/hcps-j3442688", namespace="ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="73b11fbd-18cc-490d-bc49-f492731b9079"
  Description: Pod ocm-arohcpci01-2sbhtcj4djmllishlrj1ubeicnu0g5pf-sre-hcp-cluster/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-22T01:46:18Z
  Ended:       2026-08-22T01:59:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2714112-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.77:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2714112/providers/microsoft.monitor/accounts/hcps-j2714112", namespace="ocm-arohcpci01-2sbapa1q98vqjgeok5d2vk39oquh04bn-e1r3y8g6m6m5y1i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2f5f22c6-a50a-4fc1-856c-91f66f6f3ed2"
  Description: Pod ocm-arohcpci01-2sbapa1q98vqjgeok5d2vk39oquh04bn-e1r3y8g6m6m5y1i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-08-22T01:46:18Z
  Ended:       2026-08-22T02:06:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2714112-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.77:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2714112/providers/microsoft.monitor/accounts/hcps-j2714112", namespace="ocm-arohcpci01-2sbapa1q98vqjgeok5d2vk39oquh04bn-e1r3y8g6m6m5y1i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2f5f22c6-a50a-4fc1-856c-91f66f6f3ed2"
  Description: Pod ocm-arohcpci01-2sbapa1q98vqjgeok5d2vk39oquh04bn-e1r3y8g6m6m5y1i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire9
Affected runs (9)
Date (UTC)Associated PRProw job
PR #65752091931812560375808
PR #66402091085161763442688
PR #66392090960143322714112
PR #65572090465920358027264
PR #65842090405439429677056
PR #66082090191599463043072
PR #66012090133175878553600
PR #65532089754858566455296
PR #59002089753382901256192
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert91.53%9 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 2 · Aug 18: 1 · Aug 19: 3 · Aug 20: 2 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T14:40:10Z
  Ended:       2026-08-24T14:49:11Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6438784-svc", component="backend", name="csstatedump", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-21T07:30:21Z
  Ended:       2026-08-21T07:38:24Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4553984-svc", component="backend", name="csstatedump", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-08-21T07:31:20Z
  Ended:       2026-08-21T07:38:24Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4553984-svc", component="backend", name="datadump", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-21T05:41:29Z
  Ended:       2026-08-21T05:56:28Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8412928-svc", component="backend", name="createclusterscopedreaddesires", severity="warning"
  Description: Backend controller workqueue createclusterscopedreaddesires has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-08-21T05:41:29Z
  Ended:       2026-08-21T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8412928-svc", component="backend", name="billingdump", severity="warning"
  Description: Backend controller workqueue billingdump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-08-21T05:41:29Z
  Ended:       2026-08-21T05:56:28Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8412928-svc", component="backend", name="datadump", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire9
Affected runs (9)
Date (UTC)Associated PRProw job
PR #64332091879927736438784
PR #65752090676319984553984
PR #63632090657643948412928
PR #66222090464934205853696
PR #66162090329417594179584
PR #64922090198433062195200
PR #65882090164198184062976
PR #65832090068841211105280
PR #65832089677306816630784
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e81.36%8 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsAug 17: 2 · Aug 18: 11 · Aug 19: 2 · Aug 20: 0 · Aug 21: 0 · Aug 22: 4 · Aug 23: 0 · Aug 24: 0INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc0009a11a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-s68dh8vhmfjq" to finish creating: GET https://rp.j1593344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41fcc53a-afde-489d-8eb8-76949cdd9b00
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41fcc53a-afde-489d-8eb8-76949cdd9b00",
      "name": "41fcc53a-afde-489d-8eb8-76949cdd9b00",
      "status": "Failed",
      "startTime": "2026-08-22T01:18:54.359713618Z",
      "endTime": "2026-08-22T01:38:02.522784991Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc0009a11a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-s68dh8vhmfjq" to finish creating: GET https://rp.j1593344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41fcc53a-afde-489d-8eb8-76949cdd9b00
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41fcc53a-afde-489d-8eb8-76949cdd9b00",
      "name": "41fcc53a-afde-489d-8eb8-76949cdd9b00",
      "status": "Failed",
      "startTime": "2026-08-22T01:18:54.359713618Z",
      "endTime": "2026-08-22T01:38:02.522784991Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-kbpdrprmz7mq/cluster-candidate-4-21-b4jg8c should provision
Unexpected error:
    <*fmt.wrapError | 0xc001c1c820>: 
    failed to create HCP cluster cluster-candidate-4-21-b4jg8c: failed waiting for cluster="cluster-candidate-4-21-b4jg8c" in resourcegroup="rg-candidate-4-21-kbpdrprmz7mq" to finish creating: GET https://rp.j1593344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5710b43e-bf93-4eff-a856-1083a1a7d12c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5710b43e-bf93-4eff-a856-1083a1a7d12c",
      "name": "5710b43e-bf93-4eff-a856-1083a1a7d12c",
      "status": "Failed",
      "startTime": "2026-08-22T01:17:59.540344003Z",
      "endTime": "2026-08-22T01:37:02.483270247Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-kbpdrprmz7mq/cluster-candidate-4-21-b4jg8c should provision
Unexpected error:
    <*fmt.wrapError | 0xc001c1c820>: 
    failed to create HCP cluster cluster-candidate-4-21-b4jg8c: failed waiting for cluster="cluster-candidate-4-21-b4jg8c" in resourcegroup="rg-candidate-4-21-kbpdrprmz7mq" to finish creating: GET https://rp.j1593344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5710b43e-bf93-4eff-a856-1083a1a7d12c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5710b43e-bf93-4eff-a856-1083a1a7d12c",
      "name": "5710b43e-bf93-4eff-a856-1083a1a7d12c",
      "status": "Failed",
      "startTime": "2026-08-22T01:17:59.540344003Z",
      "endTime": "2026-08-22T01:37:02.483270247Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f82620>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-h2svf5g2kxfd" to finish creating: GET https://rp.j1593344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50d4343e-33b2-4cbc-adf4-8b66762f4702
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50d4343e-33b2-4cbc-adf4-8b66762f4702",
      "name": "50d4343e-33b2-4cbc-adf4-8b66762f4702",
      "status": "Failed",
      "startTime": "2026-08-22T01:19:57.95592072Z",
      "endTime": "2026-08-22T01:39:02.502727987Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f82620>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-h2svf5g2kxfd" to finish creating: GET https://rp.j1593344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50d4343e-33b2-4cbc-adf4-8b66762f4702
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/50d4343e-33b2-4cbc-adf4-8b66762f4702",
      "name": "50d4343e-33b2-4cbc-adf4-8b66762f4702",
      "status": "Failed",
      "startTime": "2026-08-22T01:19:57.95592072Z",
      "endTime": "2026-08-22T01:39:02.502727987Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (11)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (8)
Date (UTC)Associated PRProw job
PR #65932090954745601593344
PR #65992090146594451099648
PR #64922089786820278620160
PR #45662089740517767122944
PR #65272089725920976834560
PR #65642089725877364461568
PR #65862089708355189739520
PR #65732089665224855523328
ERROR CODE: DeploymentStackDeploymentFailed
full failure pattern:
ERROR CODE: DeploymentStackDeploymentFailed
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
provision81.36%8 of 587 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsAug 17: 0 · Aug 18: 2 · Aug 19: 1 · Aug 20: 3 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
time=2026-08-21T22:06:45.798Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-08-21T22:07:37.070Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/37a71f39-392c-4236-b7bc-d450dfa4c5b6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/37a71f39-392c-4236-b7bc-d450dfa4c5b6\",\n  \"name\": \"37a71f39-392c-4236-b7bc-d450dfa4c5b6\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'f47e133e-49d1-4df1-8690-f91ad3063ea3'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4648448/providers/Microsoft.Resources/deployments/y5flko475afn22dntfyewzfj7gfkbiwdmj7ezjquufwse-260821221zuwz\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[3].Expression, Attempted value: null, Error: 1:110: parse error: unknown escape sequence U+002E '.' Activity ID: 579c60d6-5bc9-4a52-b767-c8d018c5f149.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-08-21T21:50:06.716Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-08-21T21:51:27.408Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/dadf4cf0-aee7-46d3-92d2-f8fc4f1e4cab\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/dadf4cf0-aee7-46d3-92d2-f8fc4f1e4cab\",\n  \"name\": \"dadf4cf0-aee7-46d3-92d2-f8fc4f1e4cab\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'c8f9f91e-9264-4c83-804f-b9f2716a3570'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8878848/providers/Microsoft.Resources/deployments/y5flko475afn22dntfyewzfj7gfkbiwdmj7ezjquufwse-26082121eod0o\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[3].Expression, Attempted value: null, Error: 1:110: parse error: unknown escape sequence U+002E '.' Activity ID: 64eee31e-d466-4285-abd9-d9333b56d87c.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-08-20T03:05:18.803Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-08-20T03:06:10.220Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/32d5ddb3-12f2-4f69-9adc-281d5a5aee90\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/32d5ddb3-12f2-4f69-9adc-281d5a5aee90\",\n  \"name\": \"32d5ddb3-12f2-4f69-9adc-281d5a5aee90\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'ed733f51-4922-4e52-8437-524f6699e679'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3579264/providers/Microsoft.Resources/deployments/y5flko475afn22dntfyewzfj7gfkbiwdmj7ezjquufwse-260820031kmin\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[3].Expression, Attempted value: null, Error: 1:110: parse error: unknown escape sequence U+002E '.' Activity ID: b6b6ddad-1c68-456d-966d-a061c184aaf2.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Monitoring/regional/monitoring8
Affected runs (8)
Date (UTC)Associated PRProw job
PR #65542090921650764648448
PR #65542090912921788878848
PR #65542090266981373579264
PR #65542090256601465229312
PR #65542090233522550738944
PR #66142090165593054384128
PR #65542089852729714282496
PR #65542089843308665769984
alert [svc] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [svc] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert71.19%7 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsAug 17: 2 · Aug 18: 1 · Aug 19: 3 · Aug 20: 0 · Aug 21: 1 · Aug 22: 1 · Aug 23: 0 · Aug 24: 1none
Full failure examples (3)
alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-24T18:15:41Z
  Ended:       2026-08-24T18:21:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-24T18:15:41Z
  Ended:       2026-08-24T18:22:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-08-24T18:15:41Z
  Ended:       2026-08-24T18:22:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-08-24T18:15:41Z
  Ended:       2026-08-24T18:23:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-08-24T18:15:41Z
  Ended:       2026-08-24T18:22:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-08-24T18:15:41Z
  Ended:       2026-08-24T18:23:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0375808-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.140:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0375808/providers/microsoft.monitor/accounts/services-j0375808", namespace="klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k", pod="arohcp-monitor-kube-state-metrics-5b9fdd8b5b-hf5qz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sd37mkptl2qprba4ejkvlbldgl77u9k/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-08-22T06:11:54Z
  Ended:       2026-08-22T06:23:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/services-j8029824", namespace="klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-08-22T06:11:54Z
  Ended:       2026-08-22T06:17:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/services-j8029824", namespace="klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-08-22T06:11:54Z
  Ended:       2026-08-22T06:21:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/services-j8029824", namespace="klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-08-22T06:11:54Z
  Ended:       2026-08-22T06:23:57Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/services-j8029824", namespace="klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-08-22T06:11:54Z
  Ended:       2026-08-22T06:19:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/services-j8029824", namespace="klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-08-22T06:11:54Z
  Ended:       2026-08-22T06:19:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8029824-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.144:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8029824/providers/microsoft.monitor/accounts/services-j8029824", namespace="klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-f54dn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2sbeee9sknhjmru28j861qqr85i8o22i/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T03:07:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6439552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6439552/providers/microsoft.monitor/accounts/services-j6439552", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-6t89j", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T03:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6439552-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.130:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6439552/providers/microsoft.monitor/accounts/services-j6439552", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-6t89j", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #65752091931812560375808
PR #65572091023619198029824
PR #66172090617602316439552
PR #66082090148755918557184
PR #60702090078040460627968
PR #65382089990952641892352
PR #65272089725920976834560
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] <no_message>; [clusterServiceStatus] <no_message>; ...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] <no_message>; [clusterServiceStatus] <no_message>; [descendantResources] <no_message>; [hostedCluster] <no_message>; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
e2e71.19%7 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsAug 17: 1 · Aug 18: 1 · Aug 19: 1 · Aug 20: 2 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001c982a0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-gdnhnw" in resourcegroup="rg-hs-pre--hm7prkdxbkw6" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44",
      "name": "67ed1acc-0077-4b07-848d-484f46170f44",
      "status": "Failed",
      "startTime": "2026-08-24T15:21:50.42303027Z",
      "endTime": "2026-08-24T15:45:59.872804145Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001c982a0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-gdnhnw" in resourcegroup="rg-hs-pre--hm7prkdxbkw6" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67ed1acc-0077-4b07-848d-484f46170f44",
      "name": "67ed1acc-0077-4b07-848d-484f46170f44",
      "status": "Failed",
      "startTime": "2026-08-24T15:21:50.42303027Z",
      "endTime": "2026-08-24T15:45:59.872804145Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001fe00d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-sbns7nl57fsx" to finish deleting: GET https://rp.j2704896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02c893ba-d638-4b2d-9ac6-b24d9ffc2c41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02c893ba-d638-4b2d-9ac6-b24d9ffc2c41",
      "name": "02c893ba-d638-4b2d-9ac6-b24d9ffc2c41",
      "status": "Failed",
      "startTime": "2026-08-21T05:44:31.451258644Z",
      "endTime": "2026-08-21T06:08:32.645558792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001fe00d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-sbns7nl57fsx" to finish deleting: GET https://rp.j2704896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02c893ba-d638-4b2d-9ac6-b24d9ffc2c41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02c893ba-d638-4b2d-9ac6-b24d9ffc2c41",
      "name": "02c893ba-d638-4b2d-9ac6-b24d9ffc2c41",
      "status": "Failed",
      "startTime": "2026-08-21T05:44:31.451258644Z",
      "endTime": "2026-08-21T06:08:32.645558792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001524090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-znkgn5hxfbf9" to finish deleting: GET https://rp.j3708160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fec91-844b-4c62-aa4f-0c1bd98999c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fec91-844b-4c62-aa4f-0c1bd98999c6",
      "name": "7c4fec91-844b-4c62-aa4f-0c1bd98999c6",
      "status": "Failed",
      "startTime": "2026-08-21T02:32:47.45529919Z",
      "endTime": "2026-08-21T02:56:48.748022108Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001524090>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-znkgn5hxfbf9" to finish deleting: GET https://rp.j3708160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fec91-844b-4c62-aa4f-0c1bd98999c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fec91-844b-4c62-aa4f-0c1bd98999c6",
      "name": "7c4fec91-844b-4c62-aa4f-0c1bd98999c6",
      "status": "Failed",
      "startTime": "2026-08-21T02:32:47.45529919Z",
      "endTime": "2026-08-21T02:56:48.748022108Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (7)
Date (UTC)Associated PRProw job
PR #62322091892836139012096
PR #66292090657608682704896
PR #66122090612430223708160
PR #66212090514755771961344
PR #65902090497381098328064
PR #66112090111264041734144
PR #65882089759198756737024
VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4
full failure pattern:
VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
e2e71.19%7 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsAug 17: 2 · Aug 18: 4 · Aug 19: 1 · Aug 20: 1 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:194]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011f9ad0>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21, targetMinor=4.22) failed: clusterversion status.history has no version in target minor "4.22"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:194]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011f9ad0>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21, targetMinor=4.22) failed: clusterversion status.history has no version in target minor "4.22"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f8a348>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21, targetMinor=4.22) failed: clusterversion status.history has no version in target minor "4.22"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f8a348>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21, targetMinor=4.22) failed: clusterversion status.history has no version in target minor "4.22"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc001a2ab10>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21, targetMinor=4.22) failed: clusterversion status.history has no version in target minor "4.22"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc001a2ab10>: 
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21, targetMinor=4.22) failed: clusterversion status.history has no version in target minor "4.22"
    ...
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
Affected runs (7)
Date (UTC)Associated PRProw job
PR #66122090912019128520704
PR #65842090514291462508544
PR #66012090079394612645888
PR #65882089829163719462912
PR #61572089723779814002688
PR #65852089708056525934592
PR #60702089685516600479744
alert [hcp] KubeContainerWaiting fired
full failure pattern:
alert [hcp] KubeContainerWaiting fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
alert61.02%6 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsAug 17: 4 · Aug 18: 4 · Aug 19: 2 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:14:02Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g on container healthz has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:14:03Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:14:03Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:14:03Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9r1sjk7nl7cq2gcvnm7d10nbm2ae5f-i2j7w2x8e1s9q5g on container etcd has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:15:04Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:15:04Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:15:04Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u on container etcd has been in waiting state for longer than 1 hour.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-19T20:15:04Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j1797504-mgmt-1", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9r2ec2tegi43jr5a45h7ojphgg4rhv-f0v3l6y2u5x0q0u on container healthz has been in waiting state for longer than 1 hour.
alert fired 45 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:55:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="manager", namespace="ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q", pod="capi-provider-58999f64dc-bltc4", severity="warning"
  Description: pod/capi-provider-58999f64dc-bltc4 in namespace ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q on container manager has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h on container etcd has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q", pod="etcd-0", severity="warning"
  Description: pod/etcd-0 in namespace ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q on container etcd has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q", pod="etcd-0", severity="warning"
  Description: pod/etcd-0 in namespace ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q", pod="etcd-0", severity="warning"
  Description: pod/etcd-0 in namespace ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q on container healthz has been in waiting state for longer than 1 hour.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q", pod="etcd-0", severity="warning"
  Description: pod/etcd-0 in namespace ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:56:12Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h on container healthz has been in waiting state for longer than 1 hour.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e on container healthz has been in waiting state for longer than 1 hour.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f on container etcd has been in waiting state for longer than 1 hour.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster on container etcd has been in waiting state for longer than 1 hour.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e on container etcd has been in waiting state for longer than 1 hour.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f on container healthz has been in waiting state for longer than 1 hour.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster on container healthz has been in waiting state for longer than 1 hour.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:09Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s on container healthz has been in waiting state for longer than 1 hour.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:10Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:10Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s on container etcd has been in waiting state for longer than 1 hour.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:10Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:10Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:10Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:57:10Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d on container etcd has been in waiting state for longer than 1 hour.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d on container healthz has been in waiting state for longer than 1 hour.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r on container etcd has been in waiting state for longer than 1 hour.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r on container healthz has been in waiting state for longer than 1 hour.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:58:14Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:59:19Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress on container healthz has been in waiting state for longer than 1 hour.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:59:19Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress on container etcd has been in waiting state for longer than 1 hour.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:59:19Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:59:19Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:15Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:15Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 40:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:15Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster on container healthz has been in waiting state for longer than 1 hour.
Firing 41:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:15Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 42:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:15Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster on container etcd has been in waiting state for longer than 1 hour.
Firing 43:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:15Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster on container etcd has been in waiting state for longer than 1 hour.
Firing 44:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 45:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:01:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster", pod="etcd-2", severity="warning"
  Description: pod/etcd-2 in namespace ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster on container healthz has been in waiting state for longer than 1 hour.
alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:21:36Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t on container etcd has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:21:37Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t on container etcd-metrics has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:21:37Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:21:37Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mmkm7hr956a64f4ud6162fd7al26-f4m8c9s8m7i5v9t on container healthz has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:22:35Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="etcd-defrag", namespace="ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f on container etcd-defrag has been in waiting state for longer than 1 hour.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:22:35Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="etcd", namespace="ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f on container etcd has been in waiting state for longer than 1 hour.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:22:35Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="healthz", namespace="ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f on container healthz has been in waiting state for longer than 1 hour.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:22:35Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j9462912-mgmt-1", component="kubernetes-infrastructure", container="etcd-metrics", namespace="ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f", pod="etcd-1", severity="warning"
  Description: pod/etcd-1 in namespace ocm-arohcpci01-2s98mlbgmtvccvr8mthq6lnumiqur3oo-s6y8z3g8p8j2h8f on container etcd-metrics has been in waiting state for longer than 1 hour.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeContainerWaiting does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #66112090141674171797504
PR #60702090078040460627968
PR #65882089829163719462912
PR #65832089713360550498304
PR #65852089708056525934592
PR #60702089685516600479744
expected 3 nodes to have original label key1=value1
full failure pattern:
expected 3 nodes to have original label key1=value1
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e61.02%6 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsAug 17: 0 · Aug 18: 2 · Aug 19: 1 · Aug 20: 0 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:259]: expected 3 nodes to have original label key1=value1
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:259]: expected 3 nodes to have original label key1=value1
Expected
    <bool>: ...
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #60702091855258673221632
PR #60702090817534927310848
PR #60702090766839033368576
PR #60702090078040460627968
PR #60702089685516600479744
PR #60702089644114403397632
private ingress should not be reachable from outside the VNet, but connection succeeded
full failure pattern:
private ingress should not be reachable from outside the VNet, but connection succeeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e50.85%5 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 1 · Aug 20: 1 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:186]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:186]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #66312090908436404375552
PR #65902090618825308377088
PR #66312090514791213830144
PR #65842090126361980047360
PR #65642089725877364461568
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
e2e50.85%5 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 1 · Aug 20: 1 · Aug 21: 0 · Aug 22: 1 · Aug 23: 0 · Aug 24: 1PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0010deff0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-4cmmkc" in resourcegroup="admin-credential-lifecycle-test-mk5bt6ghv29k" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7",
      "name": "0644f278-df05-4603-a10d-3a66a1f851d7",
      "status": "Failed",
      "startTime": "2026-08-24T15:17:03.672410989Z",
      "endTime": "2026-08-24T15:41:10.046088761Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tjfmep0gog9fd4pfkr5bnpc0n642 still exists (deletion dispatched at 2026-08-24T15:17:04Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0010deff0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-4cmmkc" in resourcegroup="admin-credential-lifecycle-test-mk5bt6ghv29k" to finish deleting: GET https://rp.j9012096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0644f278-df05-4603-a10d-3a66a1f851d7",
      "name": "0644f278-df05-4603-a10d-3a66a1f851d7",
      "status": "Failed",
      "startTime": "2026-08-24T15:17:03.672410989Z",
      "endTime": "2026-08-24T15:41:10.046088761Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd0tjfmep0gog9fd4pfkr5bnpc0n642 still exists (deletion dispatched at 2026-08-24T15:17:04Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0012ab8d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-khp84d8mdsh7" to finish deleting: GET https://rp.j7976064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b538a56-05be-48db-bb47-4c5a33c4accf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b538a56-05be-48db-bb47-4c5a33c4accf",
      "name": "7b538a56-05be-48db-bb47-4c5a33c4accf",
      "status": "Failed",
      "startTime": "2026-08-22T01:26:12.085661598Z",
      "endTime": "2026-08-22T01:50:12.281269064Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sbakg7dgt5f0fajefaqb6utb0vjshqh still exists (deletion dispatched at 2026-08-22T01:26:12Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0012ab8d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-khp84d8mdsh7" to finish deleting: GET https://rp.j7976064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b538a56-05be-48db-bb47-4c5a33c4accf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b538a56-05be-48db-bb47-4c5a33c4accf",
      "name": "7b538a56-05be-48db-bb47-4c5a33c4accf",
      "status": "Failed",
      "startTime": "2026-08-22T01:26:12.085661598Z",
      "endTime": "2026-08-22T01:50:12.281269064Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sbakg7dgt5f0fajefaqb6utb0vjshqh still exists (deletion dispatched at 2026-08-22T01:26:12Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0010d41e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-tr4whf2bpfsf" to finish deleting: GET https://rp.j3372032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e72e998-9d37-429a-9354-c729235c1e6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e72e998-9d37-429a-9354-c729235c1e6f",
      "name": "4e72e998-9d37-429a-9354-c729235c1e6f",
      "status": "Failed",
      "startTime": "2026-08-20T02:15:23.413243123Z",
      "endTime": "2026-08-20T02:39:29.402250383Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa13enfvdgm47tbcg2mg1bln7e4ikq7 still exists (deletion dispatched at 2026-08-20T02:15:25Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0010d41e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-tr4whf2bpfsf" to finish deleting: GET https://rp.j3372032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e72e998-9d37-429a-9354-c729235c1e6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e72e998-9d37-429a-9354-c729235c1e6f",
      "name": "4e72e998-9d37-429a-9354-c729235c1e6f",
      "status": "Failed",
      "startTime": "2026-08-20T02:15:23.413243123Z",
      "endTime": "2026-08-20T02:39:29.402250383Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa13enfvdgm47tbcg2mg1bln7e4ikq7 still exists (deletion dispatched at 2026-08-20T02:15:25Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (5)
Date (UTC)Associated PRProw job
PR #62322091892836139012096
PR #65572090958673797976064
PR #65842090249792243372032
PR #66082090087196852752384
PR #65922089796885723222016
alert [hcp] KubeDeploymentRolloutStuck fired
full failure pattern:
alert [hcp] KubeDeploymentRolloutStuck fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
alert40.68%4 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsAug 17: 4 · Aug 18: 3 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
alert fired 22 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:35:13Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h/capi-provider is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:35:13Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9na0lgat4eialup75u1revqm7m6gm8-y7z1l5m6q1x6c2h/capi-provider is not progressing for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f/capi-provider is not progressing for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s/capi-provider is not progressing for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster/capi-provider is not progressing for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e/capi-provider is not progressing for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9ift45upic3366kju9079j8v3di0-g6r2f3w3i0o4c0e/capi-provider is not progressing for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9aful10q0muufmauvdqcggd5vuq6-d9u1y3o3g5s5u1f/capi-provider is not progressing for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9lffu4hp18ss0eii14a257i26gtd-v5i9e6m0f4x4o8s/capi-provider is not progressing for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:36:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9m6u6ptoc9e5qj33o34bcjr18f0p-oidc-wi-cluster/capi-provider is not progressing for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress/capi-provider is not progressing for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d/capi-provider is not progressing for longer than 30 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9iubs7q406q106e40lr7b2sp6op1-c7x1w1p2v5m7b7d/capi-provider is not progressing for longer than 30 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9ma79lm1io9l6ulgt0sbi5a82j44-private-ingress/capi-provider is not progressing for longer than 30 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r/capi-provider is not progressing for longer than 30 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9jehqg52aoq01j4epad5srqkbk4u-q8e9e4k3l8v2m8r/capi-provider is not progressing for longer than 30 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:40:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster/capi-provider is not progressing for longer than 30 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:40:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster/capi-provider is not progressing for longer than 30 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:40:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9na9ip4t9kt4mrqougpftn4e3ok60h-cilium-cluster/capi-provider is not progressing for longer than 30 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:40:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9nampntjoikfuh2rho0ein8iunlfpp-agg-cluster/capi-provider is not progressing for longer than 30 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-08-19T15:59:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q/capi-provider is not progressing for longer than 30 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-08-19T16:00:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0627968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.202:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0627968/providers/microsoft.monitor/accounts/hcps-j0627968", namespace="ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q", pod="arohcp-monitor-kube-state-metrics-6c4c4dfcb-ctxwh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s9n9jjd3hb0tb4rfarl89p0v4d0q3ga-f8o3h2y6k4q5f9q/capi-provider is not progressing for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-18T16:58:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j4002688-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="multus-admission-controller", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4002688/providers/microsoft.monitor/accounts/hcps-j4002688", namespace="ocm-arohcpci01-2s92pk0u1ofvgc1m7g8huiv81anne9il-l7n0b6c5c7c4m8n", pod="arohcp-monitor-kube-state-metrics-74c6bc74b7-9svnw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s92pk0u1ofvgc1m7g8huiv81anne9il-l7n0b6c5c7c4m8n/multus-admission-controller is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-18T17:00:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j4002688-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="multus-admission-controller", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4002688/providers/microsoft.monitor/accounts/hcps-j4002688", namespace="ocm-arohcpci01-2s92pk0u1ofvgc1m7g8huiv81anne9il-l7n0b6c5c7c4m8n", pod="arohcp-monitor-kube-state-metrics-74c6bc74b7-9svnw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s92pk0u1ofvgc1m7g8huiv81anne9il-l7n0b6c5c7c4m8n/multus-admission-controller is not progressing for longer than 30 minutes.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-18T15:49:15Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0498304-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.171:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0498304/providers/microsoft.monitor/accounts/hcps-j0498304", namespace="ocm-arohcpci01-2s92799ecc7up4q7o2em1q0eh9d69ooa-private-kas", pod="arohcp-monitor-kube-state-metrics-74c6bc74b7-lmhjv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s92799ecc7up4q7o2em1q0eh9d69ooa-private-kas/capi-provider is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-18T15:49:15Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0498304-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.171:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0498304/providers/microsoft.monitor/accounts/hcps-j0498304", namespace="ocm-arohcpci01-2s92799ecc7up4q7o2em1q0eh9d69ooa-private-kas", pod="arohcp-monitor-kube-state-metrics-74c6bc74b7-lmhjv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s92799ecc7up4q7o2em1q0eh9d69ooa-private-kas/capi-provider is not progressing for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-18T15:49:15Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0498304-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.171:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0498304/providers/microsoft.monitor/accounts/hcps-j0498304", namespace="ocm-arohcpci01-2s926uf9b5camq56g5afnepu3udl2sh8-basic-cluster", pod="arohcp-monitor-kube-state-metrics-74c6bc74b7-lmhjv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s926uf9b5camq56g5afnepu3udl2sh8-basic-cluster/capi-provider is not progressing for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-18T15:49:15Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0498304-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.171:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0498304/providers/microsoft.monitor/accounts/hcps-j0498304", namespace="ocm-arohcpci01-2s926uf9b5camq56g5afnepu3udl2sh8-basic-cluster", pod="arohcp-monitor-kube-state-metrics-74c6bc74b7-lmhjv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2s926uf9b5camq56g5afnepu3udl2sh8-basic-cluster/capi-provider is not progressing for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentRolloutStuck does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #60702090078040460627968
PR #61572089723779814002688
PR #65832089713360550498304
PR #65852089708056525934592
alert [svc] KustoLogsDataStale fired
full failure pattern:
alert [svc] KustoLogsDataStale fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
alert40.68%4 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 1 · Aug 21: 1 · Aug 22: 0 · Aug 23: 1 · Aug 24: 0none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-23T06:20:14Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7940736-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.111:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7940736/providers/microsoft.monitor/accounts/services-j7940736", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-d5c7cc5b4-wxvwj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7940736-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-23T06:20:14Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j7940736-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.111:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7940736/providers/microsoft.monitor/accounts/services-j7940736", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-d5c7cc5b4-wxvwj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j7940736-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T14:05:51Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j2827648-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.54:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2827648/providers/microsoft.monitor/accounts/services-j2827648", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-67688f45cb-sh5gb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j2827648-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-20T21:19:06Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j2508544-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.93:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2508544/providers/microsoft.monitor/accounts/services-j2508544", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68dd4ccf65-vd77g", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j2508544-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-20T21:19:06Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j2508544-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.93:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2508544/providers/microsoft.monitor/accounts/services-j2508544", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-68dd4ccf65-vd77g", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j2508544-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #65802091370674797940736
PR #65902090767289052827648
PR #65842090514291462508544
PR #66012090079394612645888
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e40.68%4 of 587 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 2none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-fn4s2m"
Unexpected error:
    <*fmt.wrapError | 0xc000dea2e0>: 
    DNS for route host agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-fn4s2m"
Unexpected error:
    <*fmt.wrapError | 0xc000dea2e0>: 
    DNS for route host agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jhj46.apps.aro.o4q2a9i7j5z2k6t.i9i4.j4428544.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:131]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0005da4a0>: 
    DNS for route host agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:131]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0005da4a0>: 
    DNS for route host agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gr4x8.apps.aro.v20260901.3e97.j6642304.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-rp2lrf"
Unexpected error:
    <*fmt.wrapError | 0xc0009b70e0>: 
    DNS for route host agnhost-e2e-sample-app-r92l5.apps.aro.h0k3z1f6y5f5g2r.rlny.j0859904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-r92l5.apps.aro.h0k3z1f6y5f5g2r.rlny.j0859904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-r92l5.apps.aro.h0k3z1f6y5f5g2r.rlny.j0859904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-rp2lrf"
Unexpected error:
    <*fmt.wrapError | 0xc0009b70e0>: 
    DNS for route host agnhost-e2e-sample-app-r92l5.apps.aro.h0k3z1f6y5f5g2r.rlny.j0859904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-r92l5.apps.aro.h0k3z1f6y5f5g2r.rlny.j0859904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-r92l5.apps.aro.h0k3z1f6y5f5g2r.rlny.j0859904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #66292091977495384428544
PR #66182091811921836642304
PR #66012090003636900859904
PR #64262089706907987087360
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e40.68%4 of 587 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsAug 17: 1 · Aug 18: 2 · Aug 19: 1 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0012201b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-fnngq6qtshfw" to finish deleting: GET https://rp.j8029568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26314739-180a-45be-9bc0-74aedd9378b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26314739-180a-45be-9bc0-74aedd9378b7",
      "name": "26314739-180a-45be-9bc0-74aedd9378b7",
      "status": "Failed",
      "startTime": "2026-08-21T00:42:44.968602571Z",
      "endTime": "2026-08-21T01:06:45.158393217Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sakjl0nbdbh3dlb0ni55h84f7c279gs still exists (deletion dispatched at 2026-08-21T00:42:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0012201b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-fnngq6qtshfw" to finish deleting: GET https://rp.j8029568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26314739-180a-45be-9bc0-74aedd9378b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26314739-180a-45be-9bc0-74aedd9378b7",
      "name": "26314739-180a-45be-9bc0-74aedd9378b7",
      "status": "Failed",
      "startTime": "2026-08-21T00:42:44.968602571Z",
      "endTime": "2026-08-21T01:06:45.158393217Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sakjl0nbdbh3dlb0ni55h84f7c279gs still exists (deletion dispatched at 2026-08-21T00:42:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0012ae180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-jflf6brl2gcq" to finish deleting: GET https://rp.j2645888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d2b3f4-a9c9-4fe3-9d6c-919fb891cf66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d2b3f4-a9c9-4fe3-9d6c-919fb891cf66",
      "name": "b0d2b3f4-a9c9-4fe3-9d6c-919fb891cf66",
      "status": "Failed",
      "startTime": "2026-08-19T15:41:11.57390019Z",
      "endTime": "2026-08-19T16:05:19.863368819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ne21gf9tvn49fdnint8mmarsqg6fl still exists (deletion dispatched at 2026-08-19T15:41:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0012ae180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-jflf6brl2gcq" to finish deleting: GET https://rp.j2645888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d2b3f4-a9c9-4fe3-9d6c-919fb891cf66
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b0d2b3f4-a9c9-4fe3-9d6c-919fb891cf66",
      "name": "b0d2b3f4-a9c9-4fe3-9d6c-919fb891cf66",
      "status": "Failed",
      "startTime": "2026-08-19T15:41:11.57390019Z",
      "endTime": "2026-08-19T16:05:19.863368819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ne21gf9tvn49fdnint8mmarsqg6fl still exists (deletion dispatched at 2026-08-19T15:41:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001a92258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-75kk7qpk9qxd" to finish deleting: GET https://rp.j6737024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75303a92-754a-4e29-9094-d572e7177271
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75303a92-754a-4e29-9094-d572e7177271",
      "name": "75303a92-754a-4e29-9094-d572e7177271",
      "status": "Failed",
      "startTime": "2026-08-18T18:21:10.70474409Z",
      "endTime": "2026-08-18T18:45:20.134803808Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s94vo54eitg07mhl2shtkmoaum52n2q still exists (deletion dispatched at 2026-08-18T18:21:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001a92258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-75kk7qpk9qxd" to finish deleting: GET https://rp.j6737024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75303a92-754a-4e29-9094-d572e7177271
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75303a92-754a-4e29-9094-d572e7177271",
      "name": "75303a92-754a-4e29-9094-d572e7177271",
      "status": "Failed",
      "startTime": "2026-08-18T18:21:10.70474409Z",
      "endTime": "2026-08-18T18:45:20.134803808Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s94vo54eitg07mhl2shtkmoaum52n2q still exists (deletion dispatched at 2026-08-18T18:21:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #66382090580629858029568
PR #66012090079394612645888
PR #65882089759198756737024
PR #65852089708056525934592
alert [svc] KubeContainerWaiting fired
full failure pattern:
alert [svc] KubeContainerWaiting fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
alert40.68%4 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 2 · Aug 20: 1 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="collector", namespace="kube-system", pod="eraser-aks-system-31747703-vmss000000-rwxmt", severity="warning"
  Description: pod/eraser-aks-system-31747703-vmss000000-rwxmt in namespace kube-system on container collector has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="addon-token-adapter", namespace="kube-system", pod="ama-metrics-node-fnzb8", severity="warning"
  Description: pod/ama-metrics-node-fnzb8 in namespace kube-system on container addon-token-adapter has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="prometheus-collector", namespace="kube-system", pod="ama-metrics-node-fnzb8", severity="warning"
  Description: pod/ama-metrics-node-fnzb8 in namespace kube-system on container prometheus-collector has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="node-driver-registrar", namespace="kube-system", pod="aks-secrets-store-csi-driver-w8ttz", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-w8ttz in namespace kube-system on container node-driver-registrar has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="secrets-store", namespace="kube-system", pod="aks-secrets-store-csi-driver-w8ttz", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-w8ttz in namespace kube-system on container secrets-store has been in waiting state for longer than 1 hour.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="remover", namespace="kube-system", pod="eraser-aks-system-31747703-vmss000000-rwxmt", severity="warning"
  Description: pod/eraser-aks-system-31747703-vmss000000-rwxmt in namespace kube-system on container remover has been in waiting state for longer than 1 hour.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="trivy-scanner", namespace="kube-system", pod="eraser-aks-system-31747703-vmss000000-rwxmt", severity="warning"
  Description: pod/eraser-aks-system-31747703-vmss000000-rwxmt in namespace kube-system on container trivy-scanner has been in waiting state for longer than 1 hour.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-21T05:09:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j4800000-mgmt-2", component="kubernetes-infrastructure", container="liveness-probe", namespace="kube-system", pod="aks-secrets-store-csi-driver-w8ttz", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-w8ttz in namespace kube-system on container liveness-probe has been in waiting state for longer than 1 hour.
alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="node-driver-registrar", namespace="kube-system", pod="aks-secrets-store-csi-driver-724fc", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-724fc in namespace kube-system on container node-driver-registrar has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="trivy-scanner", namespace="kube-system", pod="eraser-aks-system-12106338-vmss000000-vcv9l", severity="warning"
  Description: pod/eraser-aks-system-12106338-vmss000000-vcv9l in namespace kube-system on container trivy-scanner has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="prometheus-collector", namespace="kube-system", pod="ama-metrics-node-7nn7h", severity="warning"
  Description: pod/ama-metrics-node-7nn7h in namespace kube-system on container prometheus-collector has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="secrets-store", namespace="kube-system", pod="aks-secrets-store-csi-driver-724fc", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-724fc in namespace kube-system on container secrets-store has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="liveness-probe", namespace="kube-system", pod="aks-secrets-store-csi-driver-724fc", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-724fc in namespace kube-system on container liveness-probe has been in waiting state for longer than 1 hour.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="addon-token-adapter", namespace="kube-system", pod="ama-metrics-node-7nn7h", severity="warning"
  Description: pod/ama-metrics-node-7nn7h in namespace kube-system on container addon-token-adapter has been in waiting state for longer than 1 hour.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="remover", namespace="kube-system", pod="eraser-aks-system-12106338-vmss000000-vcv9l", severity="warning"
  Description: pod/eraser-aks-system-12106338-vmss000000-vcv9l in namespace kube-system on container remover has been in waiting state for longer than 1 hour.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-20T14:40:57Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j2425216-mgmt-1", component="kubernetes-infrastructure", container="collector", namespace="kube-system", pod="eraser-aks-system-12106338-vmss000000-vcv9l", severity="warning"
  Description: pod/eraser-aks-system-12106338-vmss000000-vcv9l in namespace kube-system on container collector has been in waiting state for longer than 1 hour.
alert fired 8 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="secrets-store", namespace="kube-system", pod="aks-secrets-store-csi-driver-xlp87", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-xlp87 in namespace kube-system on container secrets-store has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="remover", namespace="kube-system", pod="eraser-aks-system-20685165-vmss000000-vp56v", severity="warning"
  Description: pod/eraser-aks-system-20685165-vmss000000-vp56v in namespace kube-system on container remover has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="collector", namespace="kube-system", pod="eraser-aks-system-20685165-vmss000000-vp56v", severity="warning"
  Description: pod/eraser-aks-system-20685165-vmss000000-vp56v in namespace kube-system on container collector has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="addon-token-adapter", namespace="kube-system", pod="ama-metrics-node-njm8m", severity="warning"
  Description: pod/ama-metrics-node-njm8m in namespace kube-system on container addon-token-adapter has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="liveness-probe", namespace="kube-system", pod="aks-secrets-store-csi-driver-xlp87", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-xlp87 in namespace kube-system on container liveness-probe has been in waiting state for longer than 1 hour.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="node-driver-registrar", namespace="kube-system", pod="aks-secrets-store-csi-driver-xlp87", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-xlp87 in namespace kube-system on container node-driver-registrar has been in waiting state for longer than 1 hour.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="trivy-scanner", namespace="kube-system", pod="eraser-aks-system-20685165-vmss000000-vp56v", severity="warning"
  Description: pod/eraser-aks-system-20685165-vmss000000-vp56v in namespace kube-system on container trivy-scanner has been in waiting state for longer than 1 hour.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-08-19T22:48:39Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6685312-mgmt-2", component="kubernetes-infrastructure", container="prometheus-collector", namespace="kube-system", pod="ama-metrics-node-njm8m", severity="warning"
  Description: pod/ama-metrics-node-njm8m in namespace kube-system on container prometheus-collector has been in waiting state for longer than 1 hour.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeContainerWaiting does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #65752090643626764800000
PR #66172090419688172425216
PR #66142090180258316685312
PR #65832090031884472946688
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
alert30.51%3 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsAug 17: 1 · Aug 18: 1 · Aug 19: 0 · Aug 20: 0 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T20:56:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3460096-mgmt-2", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-agent-666f6c69f9-jfc5v", severity="warning"
  Description: Pod maestro/maestro-agent-666f6c69f9-jfc5v has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-21T02:42:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6439552-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-7c5f797746-w2wz8", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5f797746-w2wz8 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-08-18T23:55:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0124416-mgmt-2", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-agent-6b55cfd9cc-dvzv8", severity="warning"
  Description: Pod maestro/maestro-agent-6b55cfd9cc-dvzv8 has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #66312090894760083460096
PR #66172090617602316439552
PR #61282089852994920124416
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e30.51%3 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsAug 17: 2 · Aug 18: 2 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc000a528a0>: 
    GET https://rp.j1099648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ac96a8d5-d98c-41c2-a256-36b6060bad88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ac96a8d5-d98c-41c2-a256-36b6060bad88",
      "name": "ac96a8d5-d98c-41c2-a256-36b6060bad88",
      "status": "Failed",
      "startTime": "2026-08-19T19:29:26.524288393Z",
      "endTime": "2026-08-19T19:48:29.811530415Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, ingress-operator, csi-snapshot-controller-operator, packageserver, openshift-controller-manager, olm-operator, dns-operator, cluster-storage-operator, hosted-cluster-config-operator, cluster-network-operator, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc000a528a0>: 
    GET https://rp.j1099648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ac96a8d5-d98c-41c2-a256-36b6060bad88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ac96a8d5-d98c-41c2-a256-36b6060bad88",
      "name": "ac96a8d5-d98c-41c2-a256-36b6060bad88",
      "status": "Failed",
      "startTime": "2026-08-19T19:29:26.524288393Z",
      "endTime": "2026-08-19T19:48:29.811530415Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, ingress-operator, csi-snapshot-controller-operator, packageserver, openshift-controller-manager, olm-operator, dns-operator, cluster-storage-operator, hosted-cluster-config-operator, cluster-network-operator, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-bs2lkklxljwq/cluster-candidate-4-21-hxfdst should provision
Unexpected error:
    <*fmt.wrapError | 0xc000bd7520>: 
    failed to create HCP cluster cluster-candidate-4-21-hxfdst: failed waiting for cluster="cluster-candidate-4-21-hxfdst" in resourcegroup="rg-candidate-4-21-bs2lkklxljwq" to finish creating: GET https://rp.j9343872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db06c758-c8db-4fbd-9438-a269382b691e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db06c758-c8db-4fbd-9438-a269382b691e",
      "name": "db06c758-c8db-4fbd-9438-a269382b691e",
      "status": "Failed",
      "startTime": "2026-08-18T20:19:08.987945021Z",
      "endTime": "2026-08-18T20:38:15.340514803Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, csi-snapshot-controller-operator, olm-operator, cluster-version-operator, community-operators-catalog, hosted-cluster-config-operator, ingress-operator, catalog-operator, packageserver, redhat-operators-catalog, redhat-marketplace-catalog, cluster-network-operator, dns-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-bs2lkklxljwq/cluster-candidate-4-21-hxfdst should provision
Unexpected error:
    <*fmt.wrapError | 0xc000bd7520>: 
    failed to create HCP cluster cluster-candidate-4-21-hxfdst: failed waiting for cluster="cluster-candidate-4-21-hxfdst" in resourcegroup="rg-candidate-4-21-bs2lkklxljwq" to finish creating: GET https://rp.j9343872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db06c758-c8db-4fbd-9438-a269382b691e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db06c758-c8db-4fbd-9438-a269382b691e",
      "name": "db06c758-c8db-4fbd-9438-a269382b691e",
      "status": "Failed",
      "startTime": "2026-08-18T20:19:08.987945021Z",
      "endTime": "2026-08-18T20:38:15.340514803Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, csi-snapshot-controller-operator, olm-operator, cluster-version-operator, community-operators-catalog, hosted-cluster-config-operator, ingress-operator, catalog-operator, packageserver, redhat-operators-catalog, redhat-marketplace-catalog, cluster-network-operator, dns-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000c8da60>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-q7bzmdhm7zkd" to finish creating: GET https://rp.j6278912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81876531-cc7c-4b96-8633-9c09cb6aa105
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81876531-cc7c-4b96-8633-9c09cb6aa105",
      "name": "81876531-cc7c-4b96-8633-9c09cb6aa105",
      "status": "Failed",
      "startTime": "2026-08-18T19:40:45.375485647Z",
      "endTime": "2026-08-18T19:59:46.212136225Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, catalog-operator, hosted-cluster-config-operator, cluster-network-operator, redhat-operators-catalog, redhat-marketplace-catalog, konnectivity-agent, olm-operator, certified-operators-catalog, dns-operator, ingress-operator, cluster-storage-operator, csi-snapshot-controller-operator, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000c8da60>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-q7bzmdhm7zkd" to finish creating: GET https://rp.j6278912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81876531-cc7c-4b96-8633-9c09cb6aa105
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81876531-cc7c-4b96-8633-9c09cb6aa105",
      "name": "81876531-cc7c-4b96-8633-9c09cb6aa105",
      "status": "Failed",
      "startTime": "2026-08-18T19:40:45.375485647Z",
      "endTime": "2026-08-18T19:59:46.212136225Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, catalog-operator, hosted-cluster-config-operator, cluster-network-operator, redhat-operators-catalog, redhat-marketplace-catalog, konnectivity-agent, olm-operator, certified-operators-catalog, dns-operator, ingress-operator, cluster-storage-operator, csi-snapshot-controller-operator, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #65992090146594451099648
PR #65882089797141739343872
PR #61572089786162666278912
couldn't get current server API group list: the server has asked for the client to provide credentials
full failure pattern:
couldn't get current server API group list: the server has asked for the client to provide credentials
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e30.51%3 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsAug 17: 1 · Aug 18: 1 · Aug 19: 0 · Aug 20: 1 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:347]: Timed out after 300.004s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:332 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc000c401b0>: 
    E0821 02:46:28.623242    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.657456    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.675878    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.697350    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.710066    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:347]: Timed out after 300.004s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:332 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc000c401b0>: 
    E0821 02:46:28.623242    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.657456    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.675878    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.697350    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0821 02:46:28.710066    2548 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:347]: Timed out after 300.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:332 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc001943670>: 
    E0820 10:14:05.864231    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.892161    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.910498    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.930980    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.946544    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:347]: Timed out after 300.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:332 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc001943670>: 
    E0820 10:14:05.864231    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.892161    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.910498    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.930980    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0820 10:14:05.946544    2546 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:347]: Timed out after 300.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:332 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc000f6cc00>: 
    E0818 16:01:10.950779    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:10.970180    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:10.999230    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:11.023177    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:11.041406    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:347]: Timed out after 300.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:332 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
    <*errors.errorString | 0xc000f6cc00>: 
    E0818 16:01:10.950779    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:10.970180    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:10.999230    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:11.023177    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    E0818 16:01:11.041406    2564 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
    error: You must be logged in to the server (the server has asked for the client to provide credentials)
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #66122090612430223708160
PR #65832090364783722237952
PR #61572089723779814002688
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e30.51%3 of 587 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001580bc0>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cbbn9kpl7fxv" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5",
      "name": "c6f112da-48e3-4cbb-a1b9-a47159e85ff5",
      "status": "Failed",
      "startTime": "2026-08-24T15:18:01.504032647Z",
      "endTime": "2026-08-24T15:37:08.017891025Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, community-operators-catalog, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-storage-operator, dns-operator, redhat-marketplace-catalog, cluster-version-operator, openshift-route-controller-manager, ingress-operator, kube-scheduler, cluster-image-registry-operator, konnectivity-agent, catalog-operator, cluster-network-operator, ignition-server, ignition-server-proxy, openshift-apiserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001580bc0>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-cbbn9kpl7fxv" to finish creating: GET https://rp.j1362816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c6f112da-48e3-4cbb-a1b9-a47159e85ff5",
      "name": "c6f112da-48e3-4cbb-a1b9-a47159e85ff5",
      "status": "Failed",
      "startTime": "2026-08-24T15:18:01.504032647Z",
      "endTime": "2026-08-24T15:37:08.017891025Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, azure-cloud-controller-manager, cluster-node-tuning-operator, olm-operator, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, community-operators-catalog, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-storage-operator, dns-operator, redhat-marketplace-catalog, cluster-version-operator, openshift-route-controller-manager, ingress-operator, kube-scheduler, cluster-image-registry-operator, konnectivity-agent, catalog-operator, cluster-network-operator, ignition-server, ignition-server-proxy, openshift-apiserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-ptz96v" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001880ae0>: 
    failed to create HCP cluster cluster-zstream-4-21-ptz96v: failed waiting for cluster="cluster-zstream-4-21-ptz96v" in resourcegroup="rg-zstream-upgrade-4-21-ccgv7gb5sm22" to finish creating: GET https://rp.j8125696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2fbdf9d1-a05c-4017-9c28-571dc623c672
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2fbdf9d1-a05c-4017-9c28-571dc623c672",
      "name": "2fbdf9d1-a05c-4017-9c28-571dc623c672",
      "status": "Failed",
      "startTime": "2026-08-20T18:37:36.452911298Z",
      "endTime": "2026-08-20T18:56:41.515494429Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, cluster-network-operator, certified-operators-catalog, ingress-operator, cluster-image-registry-operator, cluster-autoscaler, cluster-node-tuning-operator, dns-operator, konnectivity-agent, redhat-marketplace-catalog, kube-controller-manager, openshift-controller-manager, csi-snapshot-controller-operator, ignition-server-proxy, olm-operator, olm-collect-profiles, control-plane-pki-operator, cluster-storage-operator, catalog-operator, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, redhat-operators-catalog, machine-approver, cluster-version-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, ignition-server, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-ptz96v" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001880ae0>: 
    failed to create HCP cluster cluster-zstream-4-21-ptz96v: failed waiting for cluster="cluster-zstream-4-21-ptz96v" in resourcegroup="rg-zstream-upgrade-4-21-ccgv7gb5sm22" to finish creating: GET https://rp.j8125696.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2fbdf9d1-a05c-4017-9c28-571dc623c672
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2fbdf9d1-a05c-4017-9c28-571dc623c672",
      "name": "2fbdf9d1-a05c-4017-9c28-571dc623c672",
      "status": "Failed",
      "startTime": "2026-08-20T18:37:36.452911298Z",
      "endTime": "2026-08-20T18:56:41.515494429Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, cluster-network-operator, certified-operators-catalog, ingress-operator, cluster-image-registry-operator, cluster-autoscaler, cluster-node-tuning-operator, dns-operator, konnectivity-agent, redhat-marketplace-catalog, kube-controller-manager, openshift-controller-manager, csi-snapshot-controller-operator, ignition-server-proxy, olm-operator, olm-collect-profiles, control-plane-pki-operator, cluster-storage-operator, catalog-operator, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, redhat-operators-catalog, machine-approver, cluster-version-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, ignition-server, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000f1c4a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-kznjdfdlm5vz" to finish creating: GET https://rp.j7122944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b49e9ba3-2d03-4f2f-b0d8-a6d6a9c6f5eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b49e9ba3-2d03-4f2f-b0d8-a6d6a9c6f5eb",
      "name": "b49e9ba3-2d03-4f2f-b0d8-a6d6a9c6f5eb",
      "status": "Failed",
      "startTime": "2026-08-18T16:29:33.767295492Z",
      "endTime": "2026-08-18T16:48:39.984850743Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, dns-operator, community-operators-catalog, cluster-policy-controller, machine-approver, cluster-version-operator, openshift-controller-manager, ignition-server-proxy, olm-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, kube-scheduler, cluster-network-operator, kube-storage-version-migrator, packageserver, konnectivity-agent, ignition-server, ingress-operator, cluster-image-registry-operator, control-plane-pki-operator, kube-controller-manager, openshift-route-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, catalog-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, openshift-apiserver, redhat-marketplace-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000f1c4a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-kznjdfdlm5vz" to finish creating: GET https://rp.j7122944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b49e9ba3-2d03-4f2f-b0d8-a6d6a9c6f5eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b49e9ba3-2d03-4f2f-b0d8-a6d6a9c6f5eb",
      "name": "b49e9ba3-2d03-4f2f-b0d8-a6d6a9c6f5eb",
      "status": "Failed",
      "startTime": "2026-08-18T16:29:33.767295492Z",
      "endTime": "2026-08-18T16:48:39.984850743Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: azure-cloud-controller-manager, dns-operator, community-operators-catalog, cluster-policy-controller, machine-approver, cluster-version-operator, openshift-controller-manager, ignition-server-proxy, olm-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, kube-scheduler, cluster-network-operator, kube-storage-version-migrator, packageserver, konnectivity-agent, ignition-server, ingress-operator, cluster-image-registry-operator, control-plane-pki-operator, kube-controller-manager, openshift-route-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, catalog-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, openshift-apiserver, redhat-marketplace-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (3)
Date (UTC)Associated PRProw job
PR #65752091896822531362816
PR #65802090496257268125696
PR #45662089740517767122944
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e20.34%2 of 587 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsAug 17: 1 · Aug 18: 0 · Aug 19: 2 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 2STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e70e20>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wmkndtkx988t" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af",
      "name": "a9d4baef-549e-4561-b748-439453d113af",
      "status": "Failed",
      "startTime": "2026-08-24T15:49:08.39871397Z",
      "endTime": "2026-08-24T16:08:11.236941815Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e70e20>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wmkndtkx988t" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9d4baef-549e-4561-b748-439453d113af",
      "name": "a9d4baef-549e-4561-b748-439453d113af",
      "status": "Failed",
      "startTime": "2026-08-24T15:49:08.39871397Z",
      "endTime": "2026-08-24T16:08:11.236941815Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000408ec0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn9cdfvxcqll" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b",
      "name": "75726762-be88-402f-a564-60c91125dc8b",
      "status": "Failed",
      "startTime": "2026-08-24T15:47:28.10624954Z",
      "endTime": "2026-08-24T16:06:31.149377863Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000408ec0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn9cdfvxcqll" to finish creating: GET https://rp.j9565440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75726762-be88-402f-a564-60c91125dc8b",
      "name": "75726762-be88-402f-a564-60c91125dc8b",
      "status": "Failed",
      "startTime": "2026-08-24T15:47:28.10624954Z",
      "endTime": "2026-08-24T16:06:31.149377863Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e70880>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-x6prxbl8v6bk" to finish creating: GET https://rp.j1797504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2382ad9-6776-48ee-9bf9-e8c510952cb6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2382ad9-6776-48ee-9bf9-e8c510952cb6",
      "name": "e2382ad9-6776-48ee-9bf9-e8c510952cb6",
      "status": "Failed",
      "startTime": "2026-08-19T19:08:40.647335859Z",
      "endTime": "2026-08-19T19:27:47.028573359Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e70880>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-x6prxbl8v6bk" to finish creating: GET https://rp.j1797504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2382ad9-6776-48ee-9bf9-e8c510952cb6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2382ad9-6776-48ee-9bf9-e8c510952cb6",
      "name": "e2382ad9-6776-48ee-9bf9-e8c510952cb6",
      "status": "Failed",
      "startTime": "2026-08-19T19:08:40.647335859Z",
      "endTime": "2026-08-19T19:27:47.028573359Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #59582091901058069565440
PR #66112090141674171797504
ERROR CODE: DeploymentActive
full failure pattern:
ERROR CODE: DeploymentActive
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
provision20.34%2 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsAug 17: 1 · Aug 18: 0 · Aug 19: 1 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (2)
time=2026-08-20T12:10:57.311Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 description="Step output\n  Kind: ARM\n  Template: ./../dev-infrastructure/templates/output-mgmt.bicep\n  Parameters: ./../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam"
time=2026-08-20T12:10:57.943Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1
time=2026-08-20T12:10:58.421Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5367552-mgmt-1/providers/Microsoft.Resources/deployments/b33ca9cd40b2fdfdad4fc961f59230c1353c681efeada5d41f51b14954fa4cd6\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: DeploymentActive\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentActive\",\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5367552-mgmt-1/providers/Microsoft.Resources/deployments/b33ca9cd40b2fdfdad4fc961f59230c1353c681efeada5d41f51b14954fa4cd6' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '8/20/2026 12:10:57 PM' with correlationId '9748f509-1eae-448e-a55b-8ae0259f31d6', and will expire at '8/27/2026 12:10:57 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-08-19T01:54:26.847Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 description="Step output\n  Kind: ARM\n  Template: ../../dev-infrastructure/templates/output-mgmt.bicep\n  Parameters: ../../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam"
time=2026-08-19T01:54:27.623Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1
time=2026-08-19T01:54:28.068Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=management step=output stamp=1 err="stamp 1: failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9945216-mgmt-1/providers/Microsoft.Resources/deployments/21fa006528782108614538657b128ed438c46927450e49bcd34d09fae1d95b96\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: DeploymentActive\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"DeploymentActive\",\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9945216-mgmt-1/providers/Microsoft.Resources/deployments/21fa006528782108614538657b128ed438c46927450e49bcd34d09fae1d95b96' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '8/19/2026 1:54:27 AM' with correlationId 'b1f79be8-ea5e-4ca3-9b19-2374cbb23eb8', and will expire at '8/26/2026 1:54:27 AM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Fleet.Registration/management/output (stamp=1)1
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Velero/management/output (stamp=1)1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #65192090401415305367552
PR #65882089884673239945216
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-p...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e20.34%2 of 587 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsAug 17: 1 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 0 · Aug 24: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000f624a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-b4jk7qnglhxf" to finish creating: GET https://rp.j3592192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/034c0819-3152-42a0-b8e3-f7eba3b93869
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/034c0819-3152-42a0-b8e3-f7eba3b93869",
      "name": "034c0819-3152-42a0-b8e3-f7eba3b93869",
      "status": "Failed",
      "startTime": "2026-08-22T01:19:06.679031657Z",
      "endTime": "2026-08-22T01:38:11.645385258Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:101]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000f624a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-b4jk7qnglhxf" to finish creating: GET https://rp.j3592192.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/034c0819-3152-42a0-b8e3-f7eba3b93869
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/034c0819-3152-42a0-b8e3-f7eba3b93869",
      "name": "034c0819-3152-42a0-b8e3-f7eba3b93869",
      "status": "Failed",
      "startTime": "2026-08-22T01:19:06.679031657Z",
      "endTime": "2026-08-22T01:38:11.645385258Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-rtnpcj" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000900180>: 
    failed to create HCP cluster cluster-zstream-4-21-rtnpcj: failed waiting for cluster="cluster-zstream-4-21-rtnpcj" in resourcegroup="rg-zstream-upgrade-4-21-x7f8mnkj2vz8" to finish creating: GET https://rp.j1099648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a8aff4d-f2fe-451d-ac5c-e72fcc4613ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a8aff4d-f2fe-451d-ac5c-e72fcc4613ae",
      "name": "4a8aff4d-f2fe-451d-ac5c-e72fcc4613ae",
      "status": "Failed",
      "startTime": "2026-08-19T19:28:49.971564127Z",
      "endTime": "2026-08-19T19:47:49.803372891Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-rtnpcj" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000900180>: 
    failed to create HCP cluster cluster-zstream-4-21-rtnpcj: failed waiting for cluster="cluster-zstream-4-21-rtnpcj" in resourcegroup="rg-zstream-upgrade-4-21-x7f8mnkj2vz8" to finish creating: GET https://rp.j1099648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a8aff4d-f2fe-451d-ac5c-e72fcc4613ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a8aff4d-f2fe-451d-ac5c-e72fcc4613ae",
      "name": "4a8aff4d-f2fe-451d-ac5c-e72fcc4613ae",
      "status": "Failed",
      "startTime": "2026-08-19T19:28:49.971564127Z",
      "endTime": "2026-08-19T19:47:49.803372891Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (2)
Date (UTC)Associated PRProw job
PR #65542090958470403592192
PR #65992090146594451099648
Deserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error
e2e30.51%3 of 587 job runs affectedNoiseSignal: Noise — source deserialization/no-output errorShow trend detailsAug 17: 1 · Aug 18: 3 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
[scheduler] pool mi-containers=60 (source: LEASED_MSI_CONTAINERS), 94 specs (5×0, 87×1, 2×2+)
[scheduler] WARNING: test "Engineering should be able to retrieve kusto logs for a cluster and services" declares resource pool demands but no pool capacity is configured
  Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1787053121 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible, MIContainers:1]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
  "ts"="2026-08-18 11:38:41.084726" "level"=0 "msg"="===== TEST CASE BEGAN ====="
  "ts"="2026-08-18 11:38:41.084775" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Engineering-should-be-able-to-retrieve-kusto-logs-for-a-cluster-and-services|pid:1404"
  "ts"="2026-08-18 11:38:41.099018" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 08/18/26 11:38:41.099
  "ts"="2026-08-18 11:38:41.099181" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-n2bxlwcx6b6p"
    STEP: creating cluster parameters @ 08/18/26 11:38:43.103
    STEP: creating customer resources @ 08/18/26 11:38:43.507
    STEP: creating the HCP cluster @ 08/18/26 11:39:40.909
  "ts"="2026-08-18 11:39:40.909394" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-n2bxlwcx6b6p" "version"="4.20.34" "channelGroup"="candidate"
    STEP: verifying kusto logs are present @ 08/18/26 11:47:40.552
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb7d80d4]

goroutine 1138 [running]:
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto.(*Client).ExecutePreconfiguredQuery(0xc002209da0, {0xfb5d3d0, 0xc000dd0730}, {0xfb716a0, 0xc000fe7180}, 0xc001a8e0e0)
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto/client.go:128 +0x514
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather.(*QueryClient).ConcurrentQueries.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather/queryclient.go:68 +0x73
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1100
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95
Command Error: exit status 2
Deserialization Error: no output from command
[scheduler] pool mi-containers=60 (source: LEASED_MSI_CONTAINERS), 94 specs (5×0, 87×1, 2×2+)
[scheduler] WARNING: test "Engineering should be able to retrieve kusto logs for a cluster and services" declares resource pool demands but no pool capacity is configured
  Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1787053121 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible, MIContainers:1]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
  "ts"="2026-08-18 11:38:41.084726" "level"=0 "msg"="===== TEST CASE BEGAN ====="
  "ts"="2026-08-18 11:38:41.084775" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Engineering-should-be-able-to-retrieve-kusto-logs-for-a-cluster-and-services|pid:1404"
  "ts"="2026-08-18 11:38:41.099018" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 08/18/26 11:38:41.099
  "ts"="2026-08-18 11:38:41.099181" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-n2bxlwcx6b6p"
    STEP: creating cluster parameters @ 08/18/26 11:38:43.103
    STEP: creating customer resources @ 08/18/26 11:38:43.507
    STEP: creating the HCP cluster @ 08/18/26 11:39:40.909
  "ts"="2026-08-18 11:39:40.909394" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-n2bxlwcx6b6p" "version"="4.20.34" "channelGroup"="candidate"
    STEP: verifying kusto logs are present @ 08/18/26 11:47:40.552
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb7d80d4]

goroutine 1138 [running]:
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto.(*Client).ExecutePreconfiguredQuery(0xc002209da0, {0xfb5d3d0, 0xc000dd0730}, {0xfb716a0, 0xc000fe7180}, 0xc001a8e0e0)
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto/client.go:128 +0x514
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather.(*QueryClient).ConcurrentQueries.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather/queryclient.go:68 +0x73
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1100
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95
Command Error: exit status 2
Deserialization Error: no output from command
[scheduler] pool mi-containers=60 (source: LEASED_MSI_CONTAINERS), 94 specs (5×0, 87×1, 2×2+)
[scheduler] WARNING: test "Engineering should be able to retrieve kusto logs for a cluster and services" declares resource pool demands but no pool capacity is configured
  Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1787039952 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible, MIContainers:1]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
  "ts"="2026-08-18 07:59:12.973910" "level"=0 "msg"="===== TEST CASE BEGAN ====="
  "ts"="2026-08-18 07:59:12.973963" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Engineering-should-be-able-to-retrieve-kusto-logs-for-a-cluster-and-services|pid:1423"
  "ts"="2026-08-18 07:59:13.002203" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 08/18/26 07:59:13.002
  "ts"="2026-08-18 07:59:13.002397" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-rvz5fj9ld5qr"
    STEP: creating cluster parameters @ 08/18/26 07:59:15.168
    STEP: creating customer resources @ 08/18/26 07:59:15.469
    STEP: creating the HCP cluster @ 08/18/26 08:00:42.31
  "ts"="2026-08-18 08:00:42.310593" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-rvz5fj9ld5qr" "version"="4.20.34" "channelGroup"="candidate"
    STEP: verifying kusto logs are present @ 08/18/26 08:12:07.746
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb7d80d4]

goroutine 706 [running]:
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto.(*Client).ExecutePreconfiguredQuery(0xc0007004a0, {0xfb5d3d0, 0xc00084f400}, {0xfb716a0, 0xc000fc58c0}, 0xc00147b730)
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto/client.go:128 +0x514
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather.(*QueryClient).ConcurrentQueries.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather/queryclient.go:68 +0x73
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 685
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95
Command Error: exit status 2
Deserialization Error: no output from command
[scheduler] pool mi-containers=60 (source: LEASED_MSI_CONTAINERS), 94 specs (5×0, 87×1, 2×2+)
[scheduler] WARNING: test "Engineering should be able to retrieve kusto logs for a cluster and services" declares resource pool demands but no pool capacity is configured
  Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1787039952 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible, MIContainers:1]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
  "ts"="2026-08-18 07:59:12.973910" "level"=0 "msg"="===== TEST CASE BEGAN ====="
  "ts"="2026-08-18 07:59:12.973963" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Engineering-should-be-able-to-retrieve-kusto-logs-for-a-cluster-and-services|pid:1423"
  "ts"="2026-08-18 07:59:13.002203" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 08/18/26 07:59:13.002
  "ts"="2026-08-18 07:59:13.002397" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-rvz5fj9ld5qr"
    STEP: creating cluster parameters @ 08/18/26 07:59:15.168
    STEP: creating customer resources @ 08/18/26 07:59:15.469
    STEP: creating the HCP cluster @ 08/18/26 08:00:42.31
  "ts"="2026-08-18 08:00:42.310593" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-rvz5fj9ld5qr" "version"="4.20.34" "channelGroup"="candidate"
    STEP: verifying kusto logs are present @ 08/18/26 08:12:07.746
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb7d80d4]

goroutine 706 [running]:
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto.(*Client).ExecutePreconfiguredQuery(0xc0007004a0, {0xfb5d3d0, 0xc00084f400}, {0xfb716a0, 0xc000fc58c0}, 0xc00147b730)
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto/client.go:128 +0x514
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather.(*QueryClient).ConcurrentQueries.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather/queryclient.go:68 +0x73
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 685
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95
Command Error: exit status 2
Deserialization Error: no output from command
[scheduler] pool mi-containers=60 (source: LEASED_MSI_CONTAINERS), 94 specs (5×0, 87×1, 2×2+)
[scheduler] WARNING: test "Engineering should be able to retrieve kusto logs for a cluster and services" declares resource pool demands but no pool capacity is configured
  Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1787016532 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible, MIContainers:1]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
  "ts"="2026-08-18 01:28:52.967177" "level"=0 "msg"="===== TEST CASE BEGAN ====="
  "ts"="2026-08-18 01:28:52.967214" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Engineering-should-be-able-to-retrieve-kusto-logs-for-a-cluster-and-services|pid:1413"
  "ts"="2026-08-18 01:28:52.980139" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 08/18/26 01:28:52.98
  "ts"="2026-08-18 01:28:52.980275" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-h2xlqx47vsfl"
    STEP: creating cluster parameters @ 08/18/26 01:28:54.649
    STEP: creating customer resources @ 08/18/26 01:28:54.943
    STEP: creating the HCP cluster @ 08/18/26 01:30:02.242
  "ts"="2026-08-18 01:30:02.242888" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-h2xlqx47vsfl" "version"="4.20.34" "channelGroup"="candidate"
    STEP: verifying kusto logs are present @ 08/18/26 01:39:57.075
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb7d80d4]

goroutine 1104 [running]:
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto.(*Client).ExecutePreconfiguredQuery(0xc001be6ac0, {0xfb5d3d0, 0xc000cea640}, {0xfb716a0, 0xc000e9fb00}, 0xc0012f0460)
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto/client.go:128 +0x514
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather.(*QueryClient).ConcurrentQueries.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather/queryclient.go:68 +0x73
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1098
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95
Command Error: exit status 2
Deserialization Error: no output from command
[scheduler] pool mi-containers=60 (source: LEASED_MSI_CONTAINERS), 94 specs (5×0, 87×1, 2×2+)
[scheduler] WARNING: test "Engineering should be able to retrieve kusto logs for a cluster and services" declares resource pool demands but no pool capacity is configured
  Running Suite:  - /opt/app-root/src/github.com/Azure/ARO-HCP
  ============================================================
  Random Seed: 1787016532 - will randomize all specs

  Will run 1 of 1 specs
  ------------------------------
  Engineering should be able to retrieve kusto logs for a cluster and services [PreLaunchSetup:None, Importance:High, Positivity:Positive, Core-Infra-Service, Development-Only, ARO-HCP-RP-API-Compatible, MIContainers:1]
  /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:34
  "ts"="2026-08-18 01:28:52.967177" "level"=0 "msg"="===== TEST CASE BEGAN ====="
  "ts"="2026-08-18 01:28:52.967214" "level"=0 "msg"="Starting identity container acquisition" "count"=1 "specID"="Engineering-should-be-able-to-retrieve-kusto-logs-for-a-cluster-and-services|pid:1413"
  "ts"="2026-08-18 01:28:52.980139" "level"=0 "msg"="Successfully assigned identity containers" "count"=1 "attempt"=1 "elapsed"="0s"
    STEP: creating a resource group @ 08/18/26 01:28:52.98
  "ts"="2026-08-18 01:28:52.980275" "level"=0 "msg"="creating resource group" "resourceGroup"="kusto-logs-h2xlqx47vsfl"
    STEP: creating cluster parameters @ 08/18/26 01:28:54.649
    STEP: creating customer resources @ 08/18/26 01:28:54.943
    STEP: creating the HCP cluster @ 08/18/26 01:30:02.242
  "ts"="2026-08-18 01:30:02.242888" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="engineering-hcp-cluster" "resourceGroup"="kusto-logs-h2xlqx47vsfl" "version"="4.20.34" "channelGroup"="candidate"
    STEP: verifying kusto logs are present @ 08/18/26 01:39:57.075
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb7d80d4]

goroutine 1104 [running]:
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto.(*Client).ExecutePreconfiguredQuery(0xc001be6ac0, {0xfb5d3d0, 0xc000cea640}, {0xfb716a0, 0xc000e9fb00}, 0xc0012f0460)
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/kusto/client.go:128 +0x514
github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather.(*QueryClient).ConcurrentQueries.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/tooling/hcpctl/pkg/mustgather/queryclient.go:68 +0x73
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1098
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #65732089665224855523328
PR #65732089615938193199104
PR #65732089512043190161408

Environment: INT

Failure patterns: 12 · Failures matched: 15 · Total runs: 31 · Runs affected: 12

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
tls: failed to verify certificate: x509: certificate signed by unknown authority
full failure pattern:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e39.68%3 of 31 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsAug 17: 1 · Aug 18: 1 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-l8xph5"
Unexpected error:
    <*fmt.wrapError | 0xc000f20140>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-trf9f.apps.aro.r8g4l2d8f3g0v2i.b5du.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-l8xph5"
Unexpected error:
    <*fmt.wrapError | 0xc000f20140>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-trf9f.apps.aro.r8g4l2d8f3g0v2i.b5du.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000c8e2d0>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w89wr.apps.aro.cilium-cl.ecqe.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000c8e2d0>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w89wr.apps.aro.cilium-cl.ecqe.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:196]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0018ea198>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-jf5mq.apps.aro.cilium-cluster.exnb.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:196]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0018ea198>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-jf5mq.apps.aro.cilium-cluster.exnb.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2091735435544891392
n/a2090479680229478400
n/a2089828094020947968
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e26.45%2 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 2 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD, STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc00098d800>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-txwg4qgs4hsw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002755ed-c845-4398-a37b-8daa3b7cdc45
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002755ed-c845-4398-a37b-8daa3b7cdc45",
      "name": "002755ed-c845-4398-a37b-8daa3b7cdc45",
      "status": "Failed",
      "startTime": "2026-08-18T18:50:47.72810349Z",
      "endTime": "2026-08-18T19:09:54.253810369Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc00098d800>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-txwg4qgs4hsw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002755ed-c845-4398-a37b-8daa3b7cdc45
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002755ed-c845-4398-a37b-8daa3b7cdc45",
      "name": "002755ed-c845-4398-a37b-8daa3b7cdc45",
      "status": "Failed",
      "startTime": "2026-08-18T18:50:47.72810349Z",
      "endTime": "2026-08-18T19:09:54.253810369Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:131]: failed to create GPU nodepool gpu-np-1 with VM size Standard_NC4as_T4_v3
Unexpected error:
    <*fmt.wrapError | 0xc000a9e9c0>: 
    failed to create NodePool gpu-np-1: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-6rqw64hz9r29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df04d9fe-d188-4567-a852-79007382c2f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df04d9fe-d188-4567-a852-79007382c2f0",
      "name": "df04d9fe-d188-4567-a852-79007382c2f0",
      "status": "Failed",
      "startTime": "2026-08-18T13:10:13.902866674Z",
      "endTime": "2026-08-18T13:29:15.631640019Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:131]: failed to create GPU nodepool gpu-np-1 with VM size Standard_NC4as_T4_v3
Unexpected error:
    <*fmt.wrapError | 0xc000a9e9c0>: 
    failed to create NodePool gpu-np-1: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-6rqw64hz9r29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df04d9fe-d188-4567-a852-79007382c2f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df04d9fe-d188-4567-a852-79007382c2f0",
      "name": "df04d9fe-d188-4567-a852-79007382c2f0",
      "status": "Failed",
      "startTime": "2026-08-18T13:10:13.902866674Z",
      "endTime": "2026-08-18T13:29:15.631640019Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelHCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2089783838367551488
n/a2089682863220854784
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsAug 17: 1 · Aug 18: 1 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00101a6a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rkhdl8xqhdmz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d86f430-60d6-4c0f-b439-c3ac3403f901
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d86f430-60d6-4c0f-b439-c3ac3403f901",
      "name": "0d86f430-60d6-4c0f-b439-c3ac3403f901",
      "status": "Failed",
      "startTime": "2026-08-18T16:03:01.445081159Z",
      "endTime": "2026-08-18T16:27:09.852869865Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s92v1tvs13qsc1h7oqpk49j9bta62qs still exists (deletion dispatched at 2026-08-18T16:03:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00101a6a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rkhdl8xqhdmz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d86f430-60d6-4c0f-b439-c3ac3403f901
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0d86f430-60d6-4c0f-b439-c3ac3403f901",
      "name": "0d86f430-60d6-4c0f-b439-c3ac3403f901",
      "status": "Failed",
      "startTime": "2026-08-18T16:03:01.445081159Z",
      "endTime": "2026-08-18T16:27:09.852869865Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s92v1tvs13qsc1h7oqpk49j9bta62qs still exists (deletion dispatched at 2026-08-18T16:03:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089731977035911168
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1DEV, PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000e43900>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fbxfthcgbs8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fa0acf9-ae9d-4f99-9948-6f023a7edada
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fa0acf9-ae9d-4f99-9948-6f023a7edada",
      "name": "7fa0acf9-ae9d-4f99-9948-6f023a7edada",
      "status": "Failed",
      "startTime": "2026-08-24T10:43:39.73808157Z",
      "endTime": "2026-08-24T11:02:45.476245961Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000e43900>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fbxfthcgbs8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fa0acf9-ae9d-4f99-9948-6f023a7edada
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7fa0acf9-ae9d-4f99-9948-6f023a7edada",
      "name": "7fa0acf9-ae9d-4f99-9948-6f023a7edada",
      "status": "Failed",
      "startTime": "2026-08-24T10:43:39.73808157Z",
      "endTime": "2026-08-24T11:02:45.476245961Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2091834116805758976
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000c26120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-n4cgqv" in resourcegroup="rg-np-version-upgrade-n4cgqv-22qb5ssqb72c" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0f6000cd-5626-4ea1-84d3-e3c4775f5ae6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0f6000cd-5626-4ea1-84d3-e3c4775f5ae6",
      "name": "0f6000cd-5626-4ea1-84d3-e3c4775f5ae6",
      "status": "Failed",
      "startTime": "2026-08-19T21:24:56.829945477Z",
      "endTime": "2026-08-19T21:48:59.897240723Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9shr1s413r5943057b8kmkq36ndccb still exists (deletion dispatched at 2026-08-19T21:24:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000c26120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-n4cgqv" in resourcegroup="rg-np-version-upgrade-n4cgqv-22qb5ssqb72c" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0f6000cd-5626-4ea1-84d3-e3c4775f5ae6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0f6000cd-5626-4ea1-84d3-e3c4775f5ae6",
      "name": "0f6000cd-5626-4ea1-84d3-e3c4775f5ae6",
      "status": "Failed",
      "startTime": "2026-08-19T21:24:56.829945477Z",
      "endTime": "2026-08-19T21:48:59.897240723Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9shr1s413r5943057b8kmkq36ndccb still exists (deletion dispatched at 2026-08-19T21:24:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090162390078328832
x509: certificate signed by unknown authority
full failure pattern:
x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.000s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.000s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089731977035911168
ERROR CODE: DeploymentFailed; detail code Conflict; detail message Another operation is pending for requested object. Operation group '/operations/groups/id/|subscriptions|XXXXXXX...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code Conflict; detail message Another operation is pending for requested object. Operation group '/operations/groups/id/|subscriptions|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|resourceGroups|private-keyvault-459rvjqk6tld|privateDnsZones|privatelink.vaultcore.azure.net' already has 1 operations like '/operations/type/UpsertPrivateDnsZone/id/822c0f2c-dbd9-4336-bdd7-54dca0cd3e73_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' queued.
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:75]: failed to create customer resources with private key vault
Unexpected error:
    <*fmt.wrapError | 0xc000dee6c0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-private-kv-cluster-q9q757" in resourcegroup="private-keyvault-459rvjqk6tld" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/private-keyvault-459rvjqk6tld/providers/Microsoft.Resources/deployments/customer-infra-private-kv-cluster-q9q757/operationStatuses/08584143040338090197
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"code\": \"Conflict\",\r\n  \"message\": \"Another operation is pending for requested object. Operation group '/operations/groups/id/|subscriptions|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|resourceGroups|private-keyvault-459rvjqk6tld|privateDnsZones|privatelink.vaultcore.azure.net' already has 1 operations like '/operations/type/UpsertPrivateDnsZone/id/822c0f2c-dbd9-4336-bdd7-54dca0cd3e73_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' queued.\"\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:75]: failed to create customer resources with private key vault
Unexpected error:
    <*fmt.wrapError | 0xc000dee6c0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-private-kv-cluster-q9q757" in resourcegroup="private-keyvault-459rvjqk6tld" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/private-keyvault-459rvjqk6tld/providers/Microsoft.Resources/deployments/customer-infra-private-kv-cluster-q9q757/operationStatuses/08584143040338090197
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"code\": \"Conflict\",\r\n  \"message\": \"Another operation is pending for requested object. Operation group '/operations/groups/id/|subscriptions|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|resourceGroups|private-keyvault-459rvjqk6tld|privateDnsZones|privatelink.vaultcore.azure.net' already has 1 operations like '/operations/type/UpsertPrivateDnsZone/id/822c0f2c-dbd9-4336-bdd7-54dca0cd3e73_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' queued.\"\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090721494970667008
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000729200>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-9m9kd7rcmdds" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83f17bd3-dd33-461c-a6bb-3c1efcddd145
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83f17bd3-dd33-461c-a6bb-3c1efcddd145",
      "name": "83f17bd3-dd33-461c-a6bb-3c1efcddd145",
      "status": "Failed",
      "startTime": "2026-08-19T06:19:56.001094985Z",
      "endTime": "2026-08-19T06:38:56.416812667Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000729200>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-9m9kd7rcmdds" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83f17bd3-dd33-461c-a6bb-3c1efcddd145
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83f17bd3-dd33-461c-a6bb-3c1efcddd145",
      "name": "83f17bd3-dd33-461c-a6bb-3c1efcddd145",
      "status": "Failed",
      "startTime": "2026-08-19T06:19:56.001094985Z",
      "endTime": "2026-08-19T06:38:56.416812667Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089949305547788288
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 1 · Aug 24: 0DEV, PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-kjzq8f with version 4.21.30
Unexpected error:
    <*fmt.wrapError | 0xc0004b0bc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-kjzq8f: failed waiting for cluster="np-version-upgrade-cluster-kjzq8f" in resourcegroup="rg-np-version-upgrade-kjzq8f-kc56h5fmvk6m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dadd8ad-eeaa-4dea-bb90-c4f1255a79d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dadd8ad-eeaa-4dea-bb90-c4f1255a79d0",
      "name": "8dadd8ad-eeaa-4dea-bb90-c4f1255a79d0",
      "status": "Failed",
      "startTime": "2026-08-23T23:22:28.180803579Z",
      "endTime": "2026-08-23T23:41:32.080303176Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-kjzq8f with version 4.21.30
Unexpected error:
    <*fmt.wrapError | 0xc0004b0bc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-kjzq8f: failed waiting for cluster="np-version-upgrade-cluster-kjzq8f" in resourcegroup="rg-np-version-upgrade-kjzq8f-kc56h5fmvk6m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dadd8ad-eeaa-4dea-bb90-c4f1255a79d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8dadd8ad-eeaa-4dea-bb90-c4f1255a79d0",
      "name": "8dadd8ad-eeaa-4dea-bb90-c4f1255a79d0",
      "status": "Failed",
      "startTime": "2026-08-23T23:22:28.180803579Z",
      "endTime": "2026-08-23T23:41:32.080303176Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2091648540576059392
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": no upgrade path to update channel "<ocp-channel>" is currently available for this cluster; a channel appea...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": no upgrade path to update channel "<ocp-channel>" is currently available for this cluster; a channel appears in the cluster's desired version channels only when an upgrade edge to a release in that channel exists; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-dxvrk4" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000b99590>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-kbc8vjkst6lr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-dxvrk4
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": no upgrade path to update channel \"candidate-4.21\" is currently available for this cluster; a channel appears in the cluster's desired version channels only when an upgrade edge to a release in that channel exists",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-dxvrk4" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000b99590>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-kbc8vjkst6lr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-dxvrk4
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": no upgrade path to update channel \"candidate-4.21\" is currently available for this cluster; a channel appears in the cluster's desired version channels only when an upgrade edge to a release in that channel exists",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090639026095656960
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000693590>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20, targetMinor=4.21) failed: clusterversion status.history has no version in target minor "4.21"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000693590>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20, targetMinor=4.21) failed: clusterversion status.history has no version in target minor "4.21"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090721494970667008
failed to select node pool <nodepool> version for <ocp-channel> channel: Get "<url>": net/http: TLS handsh
full failure pattern:
failed to select node pool <nodepool> version for <ocp-channel> channel: Get "<url>": net/http: TLS handsh
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.23%1 of 31 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:207]: failed to select node pool install version for candidate-4.20 channel: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": net/http: TLS handshake timeout
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:207]: failed to select node pool install version for candidate-4.20 channel: Get "https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-4.20": net/http: TLS handshake timeout
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2091834116805758976

Environment: PROD

Failure patterns: 50 · Failures matched: 216 · Total runs: 29 · Runs affected: 22

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 8 days
e2e1551.72%15 of 29 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 8 daysShow trend detailsAug 17: 2 · Aug 18: 5 · Aug 19: 10 · Aug 20: 3 · Aug 21: 2 · Aug 22: 2 · Aug 23: 2 · Aug 24: 3DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc001f06450>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-m4qs88j7zlw8, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-m4qs88j7zlw8" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-m4qs88j7zlw8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc001f06450>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-m4qs88j7zlw8, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-m4qs88j7zlw8" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-m4qs88j7zlw8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-c58j4775b54x/cluster-nightly-5-1-rb2jwr should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001a0d800>: 
    failed to create HCP cluster cluster-nightly-5-1-rb2jwr, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-rb2jwr in resource group rg-nightly-5-1-c58j4775b54x, error: failed waiting for cluster="cluster-nightly-5-1-rb2jwr" in resourcegroup="rg-nightly-5-1-c58j4775b54x" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-rb2jwr in resource group rg-nightly-5-1-c58j4775b54x, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-c58j4775b54x/cluster-nightly-5-1-rb2jwr should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001a0d800>: 
    failed to create HCP cluster cluster-nightly-5-1-rb2jwr, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-rb2jwr in resource group rg-nightly-5-1-c58j4775b54x, error: failed waiting for cluster="cluster-nightly-5-1-rb2jwr" in resourcegroup="rg-nightly-5-1-c58j4775b54x" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-1-rb2jwr in resource group rg-nightly-5-1-c58j4775b54x, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-mqrq7j46wgxv/cluster-candidate-5-0-gm6hwh should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001400690>: 
    failed to create HCP cluster cluster-candidate-5-0-gm6hwh, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-gm6hwh in resource group rg-candidate-5-0-mqrq7j46wgxv, error: failed waiting for cluster="cluster-candidate-5-0-gm6hwh" in resourcegroup="rg-candidate-5-0-mqrq7j46wgxv" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-gm6hwh in resource group rg-candidate-5-0-mqrq7j46wgxv, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-mqrq7j46wgxv/cluster-candidate-5-0-gm6hwh should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001400690>: 
    failed to create HCP cluster cluster-candidate-5-0-gm6hwh, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-gm6hwh in resource group rg-candidate-5-0-mqrq7j46wgxv, error: failed waiting for cluster="cluster-candidate-5-0-gm6hwh" in resourcegroup="rg-candidate-5-0-mqrq7j46wgxv" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-gm6hwh in resource group rg-candidate-5-0-mqrq7j46wgxv, error: context deadline exceeded
    ...
occurred
Contributing tests (12)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public7
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.05
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.14
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.192
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
Affected runs (15)
Date (UTC)Associated PRProw job
n/a2092024382393159680
n/a2091707289294606336
n/a2091661989947379712
n/a2091344896685248512
n/a2091299562504327168
n/a2090982470148165632
n/a2090937171266506752
n/a2090619934282354688
n/a2090574634771550208
n/a2090257540720889856
n/a2090212357799153664
n/a2089991394734116864
n/a2089947767739781120
n/a2089849852371406848
n/a2089521975687909376
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e620.69%6 of 29 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsAug 17: 0 · Aug 18: 36 · Aug 19: 13 · Aug 20: 0 · Aug 21: 0 · Aug 22: 4 · Aug 23: 0 · Aug 24: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-5t5lsf5rjn4g/cluster-nightly-4-21-xmf9vj should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009ad020>: 
    failed to create HCP cluster cluster-nightly-4-21-xmf9vj: failed waiting for cluster="cluster-nightly-4-21-xmf9vj" in resourcegroup="rg-nightly-4-21-5t5lsf5rjn4g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9d217a-1f3c-416d-8de9-636190e4122d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9d217a-1f3c-416d-8de9-636190e4122d",
      "name": "5f9d217a-1f3c-416d-8de9-636190e4122d",
      "status": "Failed",
      "startTime": "2026-08-22T23:29:39.828760367Z",
      "endTime": "2026-08-22T23:48:43.906494075Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-5t5lsf5rjn4g/cluster-nightly-4-21-xmf9vj should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009ad020>: 
    failed to create HCP cluster cluster-nightly-4-21-xmf9vj: failed waiting for cluster="cluster-nightly-4-21-xmf9vj" in resourcegroup="rg-nightly-4-21-5t5lsf5rjn4g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9d217a-1f3c-416d-8de9-636190e4122d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9d217a-1f3c-416d-8de9-636190e4122d",
      "name": "5f9d217a-1f3c-416d-8de9-636190e4122d",
      "status": "Failed",
      "startTime": "2026-08-22T23:29:39.828760367Z",
      "endTime": "2026-08-22T23:48:43.906494075Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0014ce920>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-smph858sdl8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b54e40a6-a611-4bbe-8757-d1d8cc90bbc9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b54e40a6-a611-4bbe-8757-d1d8cc90bbc9",
      "name": "b54e40a6-a611-4bbe-8757-d1d8cc90bbc9",
      "status": "Failed",
      "startTime": "2026-08-22T23:04:17.915889936Z",
      "endTime": "2026-08-22T23:23:21.720195497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0014ce920>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-smph858sdl8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b54e40a6-a611-4bbe-8757-d1d8cc90bbc9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b54e40a6-a611-4bbe-8757-d1d8cc90bbc9",
      "name": "b54e40a6-a611-4bbe-8757-d1d8cc90bbc9",
      "status": "Failed",
      "startTime": "2026-08-22T23:04:17.915889936Z",
      "endTime": "2026-08-22T23:23:21.720195497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f80e40>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d2gz4r6tmvpw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c45d03f9-8ce4-4648-8394-8c6d842c5331
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c45d03f9-8ce4-4648-8394-8c6d842c5331",
      "name": "c45d03f9-8ce4-4648-8394-8c6d842c5331",
      "status": "Failed",
      "startTime": "2026-08-22T23:43:24.031623519Z",
      "endTime": "2026-08-23T00:02:33.649737296Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f80e40>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d2gz4r6tmvpw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c45d03f9-8ce4-4648-8394-8c6d842c5331
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c45d03f9-8ce4-4648-8394-8c6d842c5331",
      "name": "c45d03f9-8ce4-4648-8394-8c6d842c5331",
      "status": "Failed",
      "startTime": "2026-08-22T23:43:24.031623519Z",
      "endTime": "2026-08-23T00:02:33.649737296Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (52)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (6)
Date (UTC)Associated PRProw job
n/a2091299562504327168
n/a2089972004693217280
n/a2089901294310395904
n/a2089895148526768128
n/a2089849852371406848
n/a2089521975687909376
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e517.24%5 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 3 · Aug 20: 3 · Aug 21: 1 · Aug 22: 1 · Aug 23: 1 · Aug 24: 0DEV, INT, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc001307698>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-22-094635, targetMinor=4.22.0-0.nightly-multi-2026-08-23-080550) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-23-080550"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc001307698>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-22-094635, targetMinor=4.22.0-0.nightly-multi-2026-08-23-080550) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-23-080550"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000e06ca8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-22-094635, targetMinor=4.22.0-0.nightly-multi-2026-08-21-221014) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-21-221014"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000e06ca8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-22-094635, targetMinor=4.22.0-0.nightly-multi-2026-08-21-221014) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-21-221014"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f0d698>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-20-065923, targetMinor=4.22.0-0.nightly-multi-2026-08-20-215447) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-20-215447"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f0d698>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-20-065923, targetMinor=4.22.0-0.nightly-multi-2026-08-20-215447) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-20-215447"
    ...
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor2
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2091661989947379712
n/a2091299562504327168
n/a2090937171266506752
n/a2090574634771550208
n/a2090212357799153664
node pool <nodepool> verification failed: <node> (no version in containerRuntimeVersion "cri-o://1
full failure pattern:
node pool <nodepool> verification failed: <node> (no version in containerRuntimeVersion "cri-o://1
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e517.24%5 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 1 · Aug 22: 1 · Aug 23: 1 · Aug 24: 1STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc002b4c960>: 
    node pool upgrade verification failed: o0h1a3k2n0g4d3b-nps-420-422-9lm97-vr5xp (no version in containerRuntimeVersion "cri-o://1.35.4"); o0h1a3k2n0g4d3b-nps-420-422-9lm97-vs5vw (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"o0h1a3k2n0g4d3b-nps-420-422-9lm97-vr5xp","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"o0h1a3k2n0g4d3b-nps-420-422-9lm97-vs5vw","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc002b4c960>: 
    node pool upgrade verification failed: o0h1a3k2n0g4d3b-nps-420-422-9lm97-vr5xp (no version in containerRuntimeVersion "cri-o://1.35.4"); o0h1a3k2n0g4d3b-nps-420-422-9lm97-vs5vw (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"o0h1a3k2n0g4d3b-nps-420-422-9lm97-vr5xp","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"o0h1a3k2n0g4d3b-nps-420-422-9lm97-vs5vw","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.errorString | 0xc000515050>: 
    node pool upgrade verification failed: w1h2v3r9e6e0s7b-nps-420-422-x87c9-8vvqf (no version in containerRuntimeVersion "cri-o://1.35.4"); w1h2v3r9e6e0s7b-nps-420-422-x87c9-v4xt9 (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"w1h2v3r9e6e0s7b-nps-420-422-x87c9-8vvqf","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"w1h2v3r9e6e0s7b-nps-420-422-x87c9-v4xt9","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.errorString | 0xc000515050>: 
    node pool upgrade verification failed: w1h2v3r9e6e0s7b-nps-420-422-x87c9-8vvqf (no version in containerRuntimeVersion "cri-o://1.35.4"); w1h2v3r9e6e0s7b-nps-420-422-x87c9-v4xt9 (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"w1h2v3r9e6e0s7b-nps-420-422-x87c9-8vvqf","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"w1h2v3r9e6e0s7b-nps-420-422-x87c9-v4xt9","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.errorString | 0xc000d22130>: 
    node pool upgrade verification failed: w9t9d8g9i9t3j7r-nps-420-422-jxpcn-kdnmh (no version in containerRuntimeVersion "cri-o://1.35.4"); w9t9d8g9i9t3j7r-nps-420-422-jxpcn-zmxms (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"w9t9d8g9i9t3j7r-nps-420-422-jxpcn-kdnmh","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"w9t9d8g9i9t3j7r-nps-420-422-jxpcn-zmxms","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.errorString | 0xc000d22130>: 
    node pool upgrade verification failed: w9t9d8g9i9t3j7r-nps-420-422-jxpcn-kdnmh (no version in containerRuntimeVersion "cri-o://1.35.4"); w9t9d8g9i9t3j7r-nps-420-422-jxpcn-zmxms (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"w9t9d8g9i9t3j7r-nps-420-422-jxpcn-kdnmh","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"w9t9d8g9i9t3j7r-nps-420-422-jxpcn-zmxms","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:4eea8f361b3fcb99722b1bf436fc78b999a3ad762bd3162d9566fd6ad156d2f6","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:6e705edac6d68732998d716a353a5481340e396267372d75bf7cb6ee5b10ed45","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest5
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2092024382393159680
n/a2091661989947379712
n/a2091299562504327168
n/a2090937171266506752
n/a2090574634771550208
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e413.79%4 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 5 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 1 · Aug 24: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b1f360>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-q76sqr5mpxhb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/365f4568-cb64-4804-914b-9def1df67a45
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/365f4568-cb64-4804-914b-9def1df67a45",
      "name": "365f4568-cb64-4804-914b-9def1df67a45",
      "status": "Failed",
      "startTime": "2026-08-23T02:02:43.8249986Z",
      "endTime": "2026-08-23T02:21:49.594360516Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b1f360>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-q76sqr5mpxhb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/365f4568-cb64-4804-914b-9def1df67a45
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/365f4568-cb64-4804-914b-9def1df67a45",
      "name": "365f4568-cb64-4804-914b-9def1df67a45",
      "status": "Failed",
      "startTime": "2026-08-23T02:02:43.8249986Z",
      "endTime": "2026-08-23T02:21:49.594360516Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-zg4z5cbh25nm/cluster-candidate-4-22-tbzjpf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010dc4e0>: 
    failed to create HCP cluster cluster-candidate-4-22-tbzjpf: failed waiting for cluster="cluster-candidate-4-22-tbzjpf" in resourcegroup="rg-candidate-4-22-zg4z5cbh25nm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "name": "52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "status": "Failed",
      "startTime": "2026-08-19T07:49:18.086270762Z",
      "endTime": "2026-08-19T08:08:24.407819096Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-22-zg4z5cbh25nm/cluster-candidate-4-22-tbzjpf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010dc4e0>: 
    failed to create HCP cluster cluster-candidate-4-22-tbzjpf: failed waiting for cluster="cluster-candidate-4-22-tbzjpf" in resourcegroup="rg-candidate-4-22-zg4z5cbh25nm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "name": "52ce422b-7b70-4b90-aa52-fc3fa085301b",
      "status": "Failed",
      "startTime": "2026-08-19T07:49:18.086270762Z",
      "endTime": "2026-08-19T08:08:24.407819096Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000e87320>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-llwvfkrlbl4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "name": "c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "status": "Failed",
      "startTime": "2026-08-19T02:41:07.832237891Z",
      "endTime": "2026-08-19T03:00:11.0783774Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000e87320>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-llwvfkrlbl4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "name": "c70527cf-5589-4a4f-a61e-87d227a6cadb",
      "status": "Failed",
      "startTime": "2026-08-19T02:41:07.832237891Z",
      "endTime": "2026-08-19T03:00:11.0783774Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2091344896685248512
n/a2089972004693217280
n/a2089901294310395904
n/a2089895148526768128
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable ...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e310.34%3 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 1 · Aug 23: 1 · Aug 24: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-wzcn8d" to 5.1.0-0.nightly-multi-2026-08-21-161733
Unexpected error:
    <*fmt.wrapError | 0xc001250e60>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-wzcn8d" in resourcegroup="rg-cp-ystream-upgrade-5-1-6sv96bxs684h" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28f12e91-44fb-4368-818f-606f837a5513
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28f12e91-44fb-4368-818f-606f837a5513",
      "name": "28f12e91-44fb-4368-818f-606f837a5513",
      "status": "Failed",
      "startTime": "2026-08-23T23:47:34.702003558Z",
      "endTime": "2026-08-23T23:47:35.713542837Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-wzcn8d" to 5.1.0-0.nightly-multi-2026-08-21-161733
Unexpected error:
    <*fmt.wrapError | 0xc001250e60>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-wzcn8d" in resourcegroup="rg-cp-ystream-upgrade-5-1-6sv96bxs684h" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28f12e91-44fb-4368-818f-606f837a5513
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/28f12e91-44fb-4368-818f-606f837a5513",
      "name": "28f12e91-44fb-4368-818f-606f837a5513",
      "status": "Failed",
      "startTime": "2026-08-23T23:47:34.702003558Z",
      "endTime": "2026-08-23T23:47:35.713542837Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-8pb9fk" to 5.1.0-0.nightly-multi-2026-08-21-161733
Unexpected error:
    <*fmt.wrapError | 0xc00162c920>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-8pb9fk" in resourcegroup="rg-cp-ystream-upgrade-5-1-rvbnzc42snz2" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/983e4736-d961-45bf-8e63-27df9f4392b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/983e4736-d961-45bf-8e63-27df9f4392b0",
      "name": "983e4736-d961-45bf-8e63-27df9f4392b0",
      "status": "Failed",
      "startTime": "2026-08-22T23:48:09.603825759Z",
      "endTime": "2026-08-22T23:48:09.987134271Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-8pb9fk" to 5.1.0-0.nightly-multi-2026-08-21-161733
Unexpected error:
    <*fmt.wrapError | 0xc00162c920>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-8pb9fk" in resourcegroup="rg-cp-ystream-upgrade-5-1-rvbnzc42snz2" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/983e4736-d961-45bf-8e63-27df9f4392b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/983e4736-d961-45bf-8e63-27df9f4392b0",
      "name": "983e4736-d961-45bf-8e63-27df9f4392b0",
      "status": "Failed",
      "startTime": "2026-08-22T23:48:09.603825759Z",
      "endTime": "2026-08-22T23:48:09.987134271Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-62zzff" to 5.1.0-0.nightly-multi-2026-08-21-161733
Unexpected error:
    <*fmt.wrapError | 0xc00102a5c0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-62zzff" in resourcegroup="rg-cp-ystream-upgrade-5-1-stlzppcpw4zq" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b61f43e-2185-4f9f-9b05-3ed88171c66f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b61f43e-2185-4f9f-9b05-3ed88171c66f",
      "name": "1b61f43e-2185-4f9f-9b05-3ed88171c66f",
      "status": "Failed",
      "startTime": "2026-08-21T23:44:09.672524927Z",
      "endTime": "2026-08-21T23:44:17.756539054Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-62zzff" to 5.1.0-0.nightly-multi-2026-08-21-161733
Unexpected error:
    <*fmt.wrapError | 0xc00102a5c0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-62zzff" in resourcegroup="rg-cp-ystream-upgrade-5-1-stlzppcpw4zq" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b61f43e-2185-4f9f-9b05-3ed88171c66f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b61f43e-2185-4f9f-9b05-3ed88171c66f",
      "name": "1b61f43e-2185-4f9f-9b05-3ed88171c66f",
      "status": "Failed",
      "startTime": "2026-08-21T23:44:09.672524927Z",
      "endTime": "2026-08-21T23:44:17.756539054Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-21-184859 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor3
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2091661989947379712
n/a2091299562504327168
n/a2090937171266506752
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.35; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.35; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e310.34%3 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 1 · Aug 24: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-96fcvx" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0008e5080>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-qkzx8x2fwwlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-96fcvx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.35",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-96fcvx" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0008e5080>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-qkzx8x2fwwlv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-96fcvx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.35",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-n9x84d" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc001989aa0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-c8sxt5rwcs2x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-n9x84d
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.35",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-n9x84d" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc001989aa0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-c8sxt5rwcs2x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-n9x84d
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.35",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-bflg4k" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00170daa0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5lqcbvhjdxkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-bflg4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.35",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-bflg4k" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc00170daa0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5lqcbvhjdxkb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-bflg4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.34",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.35",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.203
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2091707289294606336
n/a2091344896685248512
n/a2090982470148165632
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.30; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.30; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e310.34%3 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 1 · Aug 24: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-znvqfb" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0008e8630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-bb24w6nmgn9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-znvqfb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.30",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-znvqfb" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0008e8630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-bb24w6nmgn9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-znvqfb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.30",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-jjmbhq" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001168f00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zv8kjsp968km/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-jjmbhq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.30",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-jjmbhq" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001168f00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zv8kjsp968km/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-jjmbhq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.30",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-v6tsw8" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000f6bc20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wd8hjklvmp49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-v6tsw8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.30",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-v6tsw8" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000f6bc20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wd8hjklvmp49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-v6tsw8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.29",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.30",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.213
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2091707289294606336
n/a2091344896685248512
n/a2090982470148165632
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.22": must be at least 4.22.10; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.22": must be at least 4.22.10; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e310.34%3 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 1 · Aug 24: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-gmlx25" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000703a40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-b6dm7hck8ksd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-gmlx25
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.10",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-gmlx25" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000703a40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-b6dm7hck8ksd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-gmlx25
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.10",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-7kr5sh" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00155f380>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-85cmmktbjpd6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-7kr5sh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.10",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-7kr5sh" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00155f380>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-85cmmktbjpd6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-7kr5sh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.10",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-hd26k2" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000914450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-99xt9th87pkp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-hd26k2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.10",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-hd26k2" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc000914450>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-99xt9th87pkp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-hd26k2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.10",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.10",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.223
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2091707289294606336
n/a2091344896685248512
n/a2090982470148165632
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5
full failure pattern:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e310.34%3 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 1 · Aug 24: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ad6720>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ad6720>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc000cde978>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc000cde978>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc001761740>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:153]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc001761740>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=5.0.0-ec.5) failed: install version "5.0.0-ec.5" not found in clusterversion/version status.history; cannot confirm the z-stream upgrade started from it
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.03
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2091707289294606336
n/a2091344896685248512
n/a2090982470148165632
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube con...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 8 · Aug 19: 40 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-21-wh4d6d" with install version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc000f13300>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-wh4d6d: failed waiting for cluster="cp-ystream-upgrade-4-21-wh4d6d" in resourcegroup="rg-cp-ystream-upgrade-4-21-jxk96xvwxvvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "name": "15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "status": "Failed",
      "startTime": "2026-08-19T09:16:07.820643421Z",
      "endTime": "2026-08-19T09:35:08.723020497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:182]: failed to create HCP cluster "cp-ystream-upgrade-4-21-wh4d6d" with install version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc000f13300>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-wh4d6d: failed waiting for cluster="cp-ystream-upgrade-4-21-wh4d6d" in resourcegroup="rg-cp-ystream-upgrade-4-21-jxk96xvwxvvc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "name": "15ccb98b-5e2b-47b1-86e9-30fb0729f99b",
      "status": "Failed",
      "startTime": "2026-08-19T09:16:07.820643421Z",
      "endTime": "2026-08-19T09:35:08.723020497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f084a0>: 
    failed to create HCP cluster np-downgrade-bb2pdp: failed waiting for cluster="np-downgrade-bb2pdp" in resourcegroup="rg-np-downgrade-bb2pdp-78j5hq9mwzxz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "name": "b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:38.906376214Z",
      "endTime": "2026-08-19T10:16:40.410448062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:593]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f084a0>: 
    failed to create HCP cluster np-downgrade-bb2pdp: failed waiting for cluster="np-downgrade-bb2pdp" in resourcegroup="rg-np-downgrade-bb2pdp-78j5hq9mwzxz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "name": "b063fb82-4df1-44ba-b542-bd61ed70bc8d",
      "status": "Failed",
      "startTime": "2026-08-19T09:57:38.906376214Z",
      "endTime": "2026-08-19T10:16:40.410448062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-gv9hgw" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d9ea20>: 
    failed to create HCP cluster cluster-zstream-4-21-gv9hgw: failed waiting for cluster="cluster-zstream-4-21-gv9hgw" in resourcegroup="rg-zstream-upgrade-4-21-5tt9ll7pp27c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898",
      "name": "72a9448e-8164-458c-aad2-98cbc2592898",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:52.253118092Z",
      "endTime": "2026-08-19T09:30:58.688039675Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: failed to create HCP cluster "cluster-zstream-4-21-gv9hgw" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d9ea20>: 
    failed to create HCP cluster cluster-zstream-4-21-gv9hgw: failed waiting for cluster="cluster-zstream-4-21-gv9hgw" in resourcegroup="rg-zstream-upgrade-4-21-5tt9ll7pp27c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/72a9448e-8164-458c-aad2-98cbc2592898",
      "name": "72a9448e-8164-458c-aad2-98cbc2592898",
      "status": "Failed",
      "startTime": "2026-08-19T09:11:52.253118092Z",
      "endTime": "2026-08-19T09:30:58.688039675Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (41)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2089991394734116864
n/a2089521975687909376
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 5 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000817120>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9k28vzcm69z8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437",
      "name": "d038a282-6d26-4349-9d6e-fafe88a9f437",
      "status": "Failed",
      "startTime": "2026-08-19T07:17:50.18783504Z",
      "endTime": "2026-08-19T07:36:53.392353745Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000817120>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9k28vzcm69z8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d038a282-6d26-4349-9d6e-fafe88a9f437",
      "name": "d038a282-6d26-4349-9d6e-fafe88a9f437",
      "status": "Failed",
      "startTime": "2026-08-19T07:17:50.18783504Z",
      "endTime": "2026-08-19T07:36:53.392353745Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-dvgp2ljncmzv/cluster-candidate-4-21-m56pdm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000866520>: 
    failed to create HCP cluster cluster-candidate-4-21-m56pdm: failed waiting for cluster="cluster-candidate-4-21-m56pdm" in resourcegroup="rg-candidate-4-21-dvgp2ljncmzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "name": "dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "status": "Failed",
      "startTime": "2026-08-19T07:48:53.852097725Z",
      "endTime": "2026-08-19T08:07:54.416992123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-21-dvgp2ljncmzv/cluster-candidate-4-21-m56pdm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000866520>: 
    failed to create HCP cluster cluster-candidate-4-21-m56pdm: failed waiting for cluster="cluster-candidate-4-21-m56pdm" in resourcegroup="rg-candidate-4-21-dvgp2ljncmzv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "name": "dc9b40e4-dff9-45e8-b9ce-61a92321163f",
      "status": "Failed",
      "startTime": "2026-08-19T07:48:53.852097725Z",
      "endTime": "2026-08-19T08:07:54.416992123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f00ec0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6k6scsrzs89w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945",
      "name": "002cf859-3aea-4235-ade3-bfe0bd472945",
      "status": "Failed",
      "startTime": "2026-08-19T02:46:18.55837028Z",
      "endTime": "2026-08-19T03:05:22.599243963Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f00ec0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-6k6scsrzs89w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/002cf859-3aea-4235-ade3-bfe0bd472945",
      "name": "002cf859-3aea-4235-ade3-bfe0bd472945",
      "status": "Failed",
      "startTime": "2026-08-19T02:46:18.55837028Z",
      "endTime": "2026-08-19T03:05:22.599243963Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2089972004693217280
n/a2089895148526768128
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:220]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:216 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2089991394734116864
n/a2089521975687909376
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-20 with version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc000706700>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-7ts9cp" in resourcegroup="rg-np-version-upgrade-7ts9cp-bztnq2hvf6sh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c93bda8-add1-4e48-9eea-0b0a433d2308
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c93bda8-add1-4e48-9eea-0b0a433d2308",
      "name": "6c93bda8-add1-4e48-9eea-0b0a433d2308",
      "status": "Failed",
      "startTime": "2026-08-21T02:54:35.924673301Z",
      "endTime": "2026-08-21T03:13:35.571507889Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-20 with version 4.20.34
Unexpected error:
    <*fmt.wrapError | 0xc000706700>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-7ts9cp" in resourcegroup="rg-np-version-upgrade-7ts9cp-bztnq2hvf6sh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c93bda8-add1-4e48-9eea-0b0a433d2308
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6c93bda8-add1-4e48-9eea-0b0a433d2308",
      "name": "6c93bda8-add1-4e48-9eea-0b0a433d2308",
      "status": "Failed",
      "startTime": "2026-08-21T02:54:35.924673301Z",
      "endTime": "2026-08-21T03:13:35.571507889Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f5dc80>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-sj5szv8t6s8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321",
      "name": "37c1aadb-f927-4c30-95d8-892451317321",
      "status": "Failed",
      "startTime": "2026-08-20T00:43:05.670558184Z",
      "endTime": "2026-08-20T01:02:11.662741691Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f5dc80>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-sj5szv8t6s8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/37c1aadb-f927-4c30-95d8-892451317321",
      "name": "37c1aadb-f927-4c30-95d8-892451317321",
      "status": "Failed",
      "startTime": "2026-08-20T00:43:05.670558184Z",
      "endTime": "2026-08-20T01:02:11.662741691Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090619934282354688
n/a2090221146522783744
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-23-080550 to 4.23.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-23-080550 to 4.23.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 1 · Aug 24: 1none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-lrdz8v" to 4.23.0-0.nightly-multi-2026-08-24-110923
Unexpected error:
    <*fmt.wrapError | 0xc000881340>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-lrdz8v" in resourcegroup="rg-cp-ystream-upgrade-4-23-t25q8kvqvdhw" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4393d587-2d85-4674-a4f6-155ceb1138f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4393d587-2d85-4674-a4f6-155ceb1138f4",
      "name": "4393d587-2d85-4674-a4f6-155ceb1138f4",
      "status": "Failed",
      "startTime": "2026-08-24T23:48:28.850341951Z",
      "endTime": "2026-08-24T23:48:30.944709051Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-23-080550 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-lrdz8v" to 4.23.0-0.nightly-multi-2026-08-24-110923
Unexpected error:
    <*fmt.wrapError | 0xc000881340>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-lrdz8v" in resourcegroup="rg-cp-ystream-upgrade-4-23-t25q8kvqvdhw" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4393d587-2d85-4674-a4f6-155ceb1138f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4393d587-2d85-4674-a4f6-155ceb1138f4",
      "name": "4393d587-2d85-4674-a4f6-155ceb1138f4",
      "status": "Failed",
      "startTime": "2026-08-24T23:48:28.850341951Z",
      "endTime": "2026-08-24T23:48:30.944709051Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-23-080550 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-fld59h" to 4.23.0-0.nightly-multi-2026-08-22-063717
Unexpected error:
    <*fmt.wrapError | 0xc0004b76c0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-fld59h" in resourcegroup="rg-cp-ystream-upgrade-4-23-rjgd8k2jwrfb" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f8fb992a-e4cf-48ed-a1c6-8971aa44e1fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f8fb992a-e4cf-48ed-a1c6-8971aa44e1fa",
      "name": "f8fb992a-e4cf-48ed-a1c6-8971aa44e1fa",
      "status": "Failed",
      "startTime": "2026-08-23T23:47:13.066383422Z",
      "endTime": "2026-08-23T23:47:20.544658196Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-23-080550 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-fld59h" to 4.23.0-0.nightly-multi-2026-08-22-063717
Unexpected error:
    <*fmt.wrapError | 0xc0004b76c0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-fld59h" in resourcegroup="rg-cp-ystream-upgrade-4-23-rjgd8k2jwrfb" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f8fb992a-e4cf-48ed-a1c6-8971aa44e1fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f8fb992a-e4cf-48ed-a1c6-8971aa44e1fa",
      "name": "f8fb992a-e4cf-48ed-a1c6-8971aa44e1fa",
      "status": "Failed",
      "startTime": "2026-08-23T23:47:13.066383422Z",
      "endTime": "2026-08-23T23:47:20.544658196Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-23-080550 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2092024382393159680
n/a2091661989947379712
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.34; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.34; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-7kjkxp" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc001506ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rncmtdm68xg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-7kjkxp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.33",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.34",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-7kjkxp" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc001506ab0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-rncmtdm68xg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-7kjkxp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.33",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.34",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-fhgsjh" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000ad93e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-khb5brts9d48/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-fhgsjh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.33",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.34",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-fhgsjh" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000ad93e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-khb5brts9d48/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-fhgsjh
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.33",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.34",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090619934282354688
n/a2090257540720889856
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.29; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.29; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-9hq8tp" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c1fa10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-dltt2s2c9z4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9hq8tp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.28",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.29",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-9hq8tp" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c1fa10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-dltt2s2c9z4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-9hq8tp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.28",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.29",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-4bcwd7" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001318120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-dcr28mz4j9bb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4bcwd7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.28",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.29",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-4bcwd7" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001318120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-dcr28mz4j9bb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4bcwd7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.28",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.29",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.212
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090619934282354688
n/a2090257540720889856
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.22": must be at least 4.22.9; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.22": must be at least 4.22.9; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e26.90%2 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-d9vwt2" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001576c60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-bmvjhbv2q2z7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-d9vwt2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.9",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.9",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-d9vwt2" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001576c60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-bmvjhbv2q2z7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-d9vwt2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.9",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.9",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-cd76bv" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0012a2480>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5z4mq62hpnnt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-cd76bv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.9",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.9",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-22-cd76bv" to minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0012a2480>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5z4mq62hpnnt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-cd76bv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": may not decrease from 4.22.9",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.22\": must be at least 4.22.9",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090619934282354688
n/a2090257540720889856
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 3 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000be4420>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-wsd5rv" in resourcegroup="rg-np-skip-minor-wsd5rv-7zc5n446c49n" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac",
      "name": "44029b45-2a06-403c-ad77-6b306bc07eac",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:10.619596164Z",
      "endTime": "2026-08-19T08:09:15.782431257Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9gasbeacg14154vgqri92i9qj3lj2u still exists (deletion dispatched at 2026-08-19T07:45:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000be4420>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-wsd5rv" in resourcegroup="rg-np-skip-minor-wsd5rv-7zc5n446c49n" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44029b45-2a06-403c-ad77-6b306bc07eac",
      "name": "44029b45-2a06-403c-ad77-6b306bc07eac",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:10.619596164Z",
      "endTime": "2026-08-19T08:09:15.782431257Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9gasbeacg14154vgqri92i9qj3lj2u still exists (deletion dispatched at 2026-08-19T07:45:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f06630>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-zm2mgg" in resourcegroup="rg-np-dg-minor-zm2mgg-fwcclg2fsc25" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "name": "655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:55.316555912Z",
      "endTime": "2026-08-19T08:09:55.778621843Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ge0h765335a8886qbg3l13s24maom still exists (deletion dispatched at 2026-08-19T07:45:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f06630>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-zm2mgg" in resourcegroup="rg-np-dg-minor-zm2mgg-fwcclg2fsc25" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "name": "655cfcac-fbdc-4612-9646-a9b1d1f8165f",
      "status": "Failed",
      "startTime": "2026-08-19T07:45:55.316555912Z",
      "endTime": "2026-08-19T08:09:55.778621843Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ge0h765335a8886qbg3l13s24maom still exists (deletion dispatched at 2026-08-19T07:45:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00116e1b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-dxqx7n" in resourcegroup="rg-np-version-upgrade-dxqx7n-j7h8sl7k4nkw" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "name": "6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "status": "Failed",
      "startTime": "2026-08-19T07:31:53.195424705Z",
      "endTime": "2026-08-19T07:55:54.488364404Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g68gmftv667t18n90kkemktfaa0e8 still exists (deletion dispatched at 2026-08-19T07:31:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00116e1b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-dxqx7n" in resourcegroup="rg-np-version-upgrade-dxqx7n-j7h8sl7k4nkw" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "name": "6d75e2fa-2f17-4c70-b06f-914c5a7c9ff5",
      "status": "Failed",
      "startTime": "2026-08-19T07:31:53.195424705Z",
      "endTime": "2026-08-19T07:55:54.488364404Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g68gmftv667t18n90kkemktfaa0e8 still exists (deletion dispatched at 2026-08-19T07:31:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089947767739781120
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 3 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00177ecc0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="pullsecret-test-7k8vgxjbrndr", caused by: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded, error: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00177ecc0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="pullsecret-test-7k8vgxjbrndr", caused by: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded, error: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7k8vgxjbrndr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster pullsecret-hcp-cluster in resource group pullsecret-test-7k8vgxjbrndr, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f46198>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm", caused by: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded, error: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000f46198>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm", caused by: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded, error: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9wbpvfcdwm" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-rm9wbpvfcdwm, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc768>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="oidc-wi-4m2nc64cmj8g", caused by: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded, error: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc768>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="oidc-wi-4m2nc64cmj8g", caused by: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded, error: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-4m2nc64cmj8g" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-4m2nc64cmj8g, error: context deadline exceeded
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089947767739781120
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degra...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00082a120>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ml5crr8fbhl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "name": "eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "status": "Failed",
      "startTime": "2026-08-19T07:18:43.866677598Z",
      "endTime": "2026-08-19T07:37:43.126289194Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc00082a120>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ml5crr8fbhl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "name": "eceba15d-5a1a-4b27-a4e7-b24724b60536",
      "status": "Failed",
      "startTime": "2026-08-19T07:18:43.866677598Z",
      "endTime": "2026-08-19T07:37:43.126289194Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089972004693217280
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc001088a60>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-9pxxtrlstzqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69f46a0a-74d3-4dcf-bf34-2997e6551fc7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69f46a0a-74d3-4dcf-bf34-2997e6551fc7",
      "name": "69f46a0a-74d3-4dcf-bf34-2997e6551fc7",
      "status": "Failed",
      "startTime": "2026-08-21T02:06:00.903480575Z",
      "endTime": "2026-08-21T02:25:03.149034019Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc001088a60>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-9pxxtrlstzqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69f46a0a-74d3-4dcf-bf34-2997e6551fc7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/69f46a0a-74d3-4dcf-bf34-2997e6551fc7",
      "name": "69f46a0a-74d3-4dcf-bf34-2997e6551fc7",
      "status": "Failed",
      "startTime": "2026-08-21T02:06:00.903480575Z",
      "endTime": "2026-08-21T02:25:03.149034019Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934282354688
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.R...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-5-0-zjc9p9k8fp8k/cluster-candidate-5-0-kz87c7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9e720>: 
    failed to create HCP cluster cluster-candidate-5-0-kz87c7: failed waiting for cluster="cluster-candidate-5-0-kz87c7" in resourcegroup="rg-candidate-5-0-zjc9p9k8fp8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "name": "a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "status": "Failed",
      "startTime": "2026-08-19T02:38:11.46423025Z",
      "endTime": "2026-08-19T02:57:21.467415333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-5-0-zjc9p9k8fp8k/cluster-candidate-5-0-kz87c7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9e720>: 
    failed to create HCP cluster cluster-candidate-5-0-kz87c7: failed waiting for cluster="cluster-candidate-5-0-kz87c7" in resourcegroup="rg-candidate-5-0-zjc9p9k8fp8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "name": "a3081846-fb1d-49f7-9bc5-d276f1e7c8f7",
      "status": "Failed",
      "startTime": "2026-08-19T02:38:11.46423025Z",
      "endTime": "2026-08-19T02:57:21.467415333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089895148526768128
tls: failed to verify certificate: x509: certificate signed by unknown authority
full failure pattern:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0INT, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000d24198>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-xjt5s.apps.aro.cilium-cl.j5nd.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000d24198>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-xjt5s.apps.aro.cilium-cl.j5nd.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089895148526768128
Cluster provisioning failed
full failure pattern:
Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089521975687909376
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000923f40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-886kmmqnwq8r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292",
      "name": "a873ac5e-6930-421b-80c5-b0073727d292",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:10.761261438Z",
      "endTime": "2026-08-19T02:22:19.897779073Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000923f40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-886kmmqnwq8r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a873ac5e-6930-421b-80c5-b0073727d292",
      "name": "a873ac5e-6930-421b-80c5-b0073727d292",
      "status": "Failed",
      "startTime": "2026-08-19T02:03:10.761261438Z",
      "endTime": "2026-08-19T02:22:19.897779073Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089895148526768128
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011f0378>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dw5mg9lsjg9t" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "name": "52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "status": "Failed",
      "startTime": "2026-08-19T07:21:20.266749938Z",
      "endTime": "2026-08-19T07:45:24.43604152Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g587oc7gqeeckl8k5s1283accvgo6 still exists (deletion dispatched at 2026-08-19T07:21:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011f0378>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-dw5mg9lsjg9t" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "name": "52d1d0b5-d84a-422f-adff-632e7b15bd86",
      "status": "Failed",
      "startTime": "2026-08-19T07:21:20.266749938Z",
      "endTime": "2026-08-19T07:45:24.43604152Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g587oc7gqeeckl8k5s1283accvgo6 still exists (deletion dispatched at 2026-08-19T07:21:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089947767739781120
Cluster should become ready within '20.000000' minutes
full failure pattern:
Cluster should become ready within '20.000000' minutes
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:163]: Timed out after 1200.003s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:163]: Timed out after 1200.003s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089991394734116864
ERROR CODE: Conflict; detail message Cannot update resource while resource is "provisioning"; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: Conflict; detail message Cannot update resource while resource is "provisioning"; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-chjtcz
Unexpected error:
    <*fmt.wrapError | 0xc000fcc1e0>: 
    failed to create HCP cluster np-noedge-chjtcz: failed starting cluster creation "np-noedge-chjtcz" in resourcegroup="rg-np-noedge-chjtcz-4lgwg5nz5hfm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:314]: failed to create HCP cluster np-noedge-chjtcz
Unexpected error:
    <*fmt.wrapError | 0xc000fcc1e0>: 
    failed to create HCP cluster np-noedge-chjtcz: failed starting cluster creation "np-noedge-chjtcz" in resourcegroup="rg-np-noedge-chjtcz-4lgwg5nz5hfm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-chjtcz-4lgwg5nz5hfm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-chjtcz"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089991394734116864
ERROR CODE: DeploymentFailed; detail code OK
full failure pattern:
ERROR CODE: DeploymentFailed; detail code OK
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000a92d60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-n784sz" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-n784sz/operationStatuses/08584144855033284603
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "OK",
            "message": ""
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000a92d60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-n784sz" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-n784sz/operationStatuses/08584144855033284603
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "OK",
            "message": ""
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089947767739781120
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-20-fkndtwz8x29s/cluster-candidate-4-20-kpswjg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c0ef80>: 
    failed to create HCP cluster cluster-candidate-4-20-kpswjg: failed waiting for cluster="cluster-candidate-4-20-kpswjg" in resourcegroup="rg-candidate-4-20-fkndtwz8x29s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "name": "5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "status": "Failed",
      "startTime": "2026-08-19T02:36:44.929925762Z",
      "endTime": "2026-08-19T02:55:50.765263478Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:107]: HCP cluster rg-candidate-4-20-fkndtwz8x29s/cluster-candidate-4-20-kpswjg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c0ef80>: 
    failed to create HCP cluster cluster-candidate-4-20-kpswjg: failed waiting for cluster="cluster-candidate-4-20-kpswjg" in resourcegroup="rg-candidate-4-20-fkndtwz8x29s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "name": "5ce8c6d6-e8b9-4d76-97a6-d8ba04a44b30",
      "status": "Failed",
      "startTime": "2026-08-19T02:36:44.929925762Z",
      "endTime": "2026-08-19T02:55:50.765263478Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089895148526768128
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000e46720>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-mj9b89zp6qrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "name": "764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "status": "Failed",
      "startTime": "2026-08-19T05:36:20.303691403Z",
      "endTime": "2026-08-19T05:55:29.476993829Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000e46720>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-mj9b89zp6qrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "name": "764a11ed-2f54-474f-bdce-7ac64c3a7a20",
      "status": "Failed",
      "startTime": "2026-08-19T05:36:20.303691403Z",
      "endTime": "2026-08-19T05:55:29.476993829Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089939212420780032
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-p...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f896e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fh79ld4nbj75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b",
      "name": "5bea30a8-9570-45db-b908-4c4e07deca9b",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:30.019566976Z",
      "endTime": "2026-08-19T07:35:32.987525861Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000f896e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fh79ld4nbj75" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5bea30a8-9570-45db-b908-4c4e07deca9b",
      "name": "5bea30a8-9570-45db-b908-4c4e07deca9b",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:30.019566976Z",
      "endTime": "2026-08-19T07:35:32.987525861Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089972004693217280
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00161c860>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-x98pjvvxgzk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "name": "03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:18.329159963Z",
      "endTime": "2026-08-19T07:35:23.12849681Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:103]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00161c860>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-x98pjvvxgzk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "name": "03e64f83-5bd0-4f0b-819b-4b88633c17cf",
      "status": "Failed",
      "startTime": "2026-08-19T07:16:18.329159963Z",
      "endTime": "2026-08-19T07:35:23.12849681Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089972004693217280
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; prov...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00091a9c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nzcg89cnmszj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "name": "99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:34.622468609Z",
      "endTime": "2026-08-19T03:01:41.644884055Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00091a9c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-nzcg89cnmszj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "name": "99a00b91-6d0c-48a1-9fdd-2970d104a1e6",
      "status": "Failed",
      "startTime": "2026-08-19T02:42:34.622468609Z",
      "endTime": "2026-08-19T03:01:41.644884055Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089901294310395904
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc8e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-85gsqcc2zjlm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "name": "102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "status": "Failed",
      "startTime": "2026-08-19T07:41:39.807626461Z",
      "endTime": "2026-08-19T08:05:45.761898963Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9geiu25jubct6f4b5k5ucmu6fmf5pl still exists (deletion dispatched at 2026-08-19T07:41:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cdc8e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-85gsqcc2zjlm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "name": "102c7ad9-cfd7-4fbb-b4af-47ae501c8b59",
      "status": "Failed",
      "startTime": "2026-08-19T07:41:39.807626461Z",
      "endTime": "2026-08-19T08:05:45.761898963Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9geiu25jubct6f4b5k5ucmu6fmf5pl still exists (deletion dispatched at 2026-08-19T07:41:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089947767739781120
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000abefa8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-9b6mxq" in resourcegroup="rg-np-noedge-9b6mxq-8gg2s6c8ddr4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "name": "fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "status": "Failed",
      "startTime": "2026-08-19T07:11:47.482159856Z",
      "endTime": "2026-08-19T07:35:54.369742847Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g7tdod2lgoa5os2e9n6onp6vt2m2l still exists (deletion dispatched at 2026-08-19T07:11:48Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000abefa8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-9b6mxq" in resourcegroup="rg-np-noedge-9b6mxq-8gg2s6c8ddr4" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "name": "fc763824-a7c1-4c84-ad4a-52a04db780c3",
      "status": "Failed",
      "startTime": "2026-08-19T07:11:47.482159856Z",
      "endTime": "2026-08-19T07:35:54.369742847Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9g7tdod2lgoa5os2e9n6onp6vt2m2l still exists (deletion dispatched at 2026-08-19T07:11:48Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089947767739781120
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-21-140538 to 4.21.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-21-140538 to 4.21.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-ltpqhc" to 4.21.0-0.nightly-multi-2026-08-20-065923
Unexpected error:
    <*fmt.wrapError | 0xc00071a3e0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-ltpqhc" in resourcegroup="rg-cp-ystream-upgrade-4-21-c4ngqczk4ph5" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/80cf611d-a2f7-4a43-b5f4-cb99c877bca2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/80cf611d-a2f7-4a43-b5f4-cb99c877bca2",
      "name": "80cf611d-a2f7-4a43-b5f4-cb99c877bca2",
      "status": "Failed",
      "startTime": "2026-08-21T23:42:45.54327206Z",
      "endTime": "2026-08-21T23:42:47.666030908Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-21-140538 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-ltpqhc" to 4.21.0-0.nightly-multi-2026-08-20-065923
Unexpected error:
    <*fmt.wrapError | 0xc00071a3e0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-ltpqhc" in resourcegroup="rg-cp-ystream-upgrade-4-21-c4ngqczk4ph5" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/80cf611d-a2f7-4a43-b5f4-cb99c877bca2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/80cf611d-a2f7-4a43-b5f4-cb99c877bca2",
      "name": "80cf611d-a2f7-4a43-b5f4-cb99c877bca2",
      "status": "Failed",
      "startTime": "2026-08-21T23:42:45.54327206Z",
      "endTime": "2026-08-21T23:42:47.666030908Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-21-140538 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090937171266506752
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-22-032601 to 4.21.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-22-032601 to 4.21.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-ljjplf" to 4.21.0-0.nightly-multi-2026-08-22-094635
Unexpected error:
    <*fmt.wrapError | 0xc000e00d00>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-ljjplf" in resourcegroup="rg-cp-ystream-upgrade-4-21-rjbgq4lgq7qt" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14160601-762a-4228-a29b-548613a97ff8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14160601-762a-4228-a29b-548613a97ff8",
      "name": "14160601-762a-4228-a29b-548613a97ff8",
      "status": "Failed",
      "startTime": "2026-08-22T23:46:44.80771206Z",
      "endTime": "2026-08-22T23:46:53.483788723Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-22-032601 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-ljjplf" to 4.21.0-0.nightly-multi-2026-08-22-094635
Unexpected error:
    <*fmt.wrapError | 0xc000e00d00>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-ljjplf" in resourcegroup="rg-cp-ystream-upgrade-4-21-rjbgq4lgq7qt" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14160601-762a-4228-a29b-548613a97ff8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/14160601-762a-4228-a29b-548613a97ff8",
      "name": "14160601-762a-4228-a29b-548613a97ff8",
      "status": "Failed",
      "startTime": "2026-08-22T23:46:44.80771206Z",
      "endTime": "2026-08-22T23:46:53.483788723Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-22-032601 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2091299562504327168
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-23-070610 to 4.21.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-23-070610 to 4.21.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 1 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-wxm9hm" to 4.21.0-0.nightly-multi-2026-08-22-094635
Unexpected error:
    <*fmt.wrapError | 0xc000a7c6c0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-wxm9hm" in resourcegroup="rg-cp-ystream-upgrade-4-21-m4n2k2rcnqcz" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ada4714-891c-4466-827a-ecc32dfa2442
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ada4714-891c-4466-827a-ecc32dfa2442",
      "name": "1ada4714-891c-4466-827a-ecc32dfa2442",
      "status": "Failed",
      "startTime": "2026-08-23T23:44:35.120120083Z",
      "endTime": "2026-08-23T23:44:40.624916223Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-23-070610 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-wxm9hm" to 4.21.0-0.nightly-multi-2026-08-22-094635
Unexpected error:
    <*fmt.wrapError | 0xc000a7c6c0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-wxm9hm" in resourcegroup="rg-cp-ystream-upgrade-4-21-m4n2k2rcnqcz" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ada4714-891c-4466-827a-ecc32dfa2442
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ada4714-891c-4466-827a-ecc32dfa2442",
      "name": "1ada4714-891c-4466-827a-ecc32dfa2442",
      "status": "Failed",
      "startTime": "2026-08-23T23:44:35.120120083Z",
      "endTime": "2026-08-23T23:44:40.624916223Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-23-070610 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2091661989947379712
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-24-135049 to 4.21.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-24-135049 to 4.21.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-7z565t" to 4.21.0-0.nightly-multi-2026-08-24-143120
Unexpected error:
    <*fmt.wrapError | 0xc0009b5de0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-7z565t" in resourcegroup="rg-cp-ystream-upgrade-4-21-98477z6czjf7" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d58c2718-67c9-46eb-afc9-3f15d71ef7a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d58c2718-67c9-46eb-afc9-3f15d71ef7a2",
      "name": "d58c2718-67c9-46eb-afc9-3f15d71ef7a2",
      "status": "Failed",
      "startTime": "2026-08-24T23:45:23.595556181Z",
      "endTime": "2026-08-24T23:45:30.682388649Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-24-135049 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-7z565t" to 4.21.0-0.nightly-multi-2026-08-24-143120
Unexpected error:
    <*fmt.wrapError | 0xc0009b5de0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-21-7z565t" in resourcegroup="rg-cp-ystream-upgrade-4-21-98477z6czjf7" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d58c2718-67c9-46eb-afc9-3f15d71ef7a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d58c2718-67c9-46eb-afc9-3f15d71ef7a2",
      "name": "d58c2718-67c9-46eb-afc9-3f15d71ef7a2",
      "status": "Failed",
      "startTime": "2026-08-24T23:45:23.595556181Z",
      "endTime": "2026-08-24T23:45:30.682388649Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.20.0-0.nightly-multi-2026-08-24-135049 to 4.21.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2092024382393159680
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.21.0-0.nightly-multi-2026-08-24-143120 to 4.22.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.21.0-0.nightly-multi-2026-08-24-143120 to 4.22.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-cbj2dm" to 4.22.0-0.nightly-multi-2026-08-23-080550
Unexpected error:
    <*fmt.wrapError | 0xc000df2720>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-22-cbj2dm" in resourcegroup="rg-cp-ystream-upgrade-4-22-4ls4ksxqchgg" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1a50e80-0d7c-4afa-9fa8-42a36f49b9c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1a50e80-0d7c-4afa-9fa8-42a36f49b9c7",
      "name": "d1a50e80-0d7c-4afa-9fa8-42a36f49b9c7",
      "status": "Failed",
      "startTime": "2026-08-24T23:48:14.416516082Z",
      "endTime": "2026-08-24T23:48:20.787265431Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.21.0-0.nightly-multi-2026-08-24-143120 to 4.22.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-cbj2dm" to 4.22.0-0.nightly-multi-2026-08-23-080550
Unexpected error:
    <*fmt.wrapError | 0xc000df2720>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-22-cbj2dm" in resourcegroup="rg-cp-ystream-upgrade-4-22-4ls4ksxqchgg" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1a50e80-0d7c-4afa-9fa8-42a36f49b9c7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d1a50e80-0d7c-4afa-9fa8-42a36f49b9c7",
      "name": "d1a50e80-0d7c-4afa-9fa8-42a36f49b9c7",
      "status": "Failed",
      "startTime": "2026-08-24T23:48:14.416516082Z",
      "endTime": "2026-08-24T23:48:20.787265431Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.21.0-0.nightly-multi-2026-08-24-143120 to 4.22.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2092024382393159680
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d5pz9g" to 4.23.0-0.nightly-multi-2026-08-19-140455
Unexpected error:
    <*fmt.wrapError | 0xc000900d00>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-d5pz9g" in resourcegroup="rg-cp-ystream-upgrade-4-23-ttgbbn52wj5b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "name": "f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "status": "Failed",
      "startTime": "2026-08-19T23:46:03.25868454Z",
      "endTime": "2026-08-19T23:46:13.75881135Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-d5pz9g" to 4.23.0-0.nightly-multi-2026-08-19-140455
Unexpected error:
    <*fmt.wrapError | 0xc000900d00>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-d5pz9g" in resourcegroup="rg-cp-ystream-upgrade-4-23-ttgbbn52wj5b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "name": "f80b855b-6ddb-4d41-bcfd-69a629fa5f3a",
      "status": "Failed",
      "startTime": "2026-08-19T23:46:03.25868454Z",
      "endTime": "2026-08-19T23:46:13.75881135Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090212357799153664
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-141954 to 4.23.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-141954 to 4.23.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-r7msf7" to 4.23.0-0.nightly-multi-2026-08-20-085020
Unexpected error:
    <*fmt.wrapError | 0xc000db4400>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-r7msf7" in resourcegroup="rg-cp-ystream-upgrade-4-23-cxr6vwskbzqs" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cbae144c-9ad6-4399-8e1e-708ba5343570
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cbae144c-9ad6-4399-8e1e-708ba5343570",
      "name": "cbae144c-9ad6-4399-8e1e-708ba5343570",
      "status": "Failed",
      "startTime": "2026-08-20T23:45:35.456682564Z",
      "endTime": "2026-08-20T23:45:36.715219027Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-141954 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-r7msf7" to 4.23.0-0.nightly-multi-2026-08-20-085020
Unexpected error:
    <*fmt.wrapError | 0xc000db4400>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-r7msf7" in resourcegroup="rg-cp-ystream-upgrade-4-23-cxr6vwskbzqs" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cbae144c-9ad6-4399-8e1e-708ba5343570
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cbae144c-9ad6-4399-8e1e-708ba5343570",
      "name": "cbae144c-9ad6-4399-8e1e-708ba5343570",
      "status": "Failed",
      "startTime": "2026-08-20T23:45:35.456682564Z",
      "endTime": "2026-08-20T23:45:36.715219027Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-141954 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090574634771550208
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-215447 to 4.23.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-215447 to 4.23.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-7jwrlb" to 4.23.0-0.nightly-multi-2026-08-21-150158
Unexpected error:
    <*fmt.wrapError | 0xc00071fec0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-7jwrlb" in resourcegroup="rg-cp-ystream-upgrade-4-23-ddtbb45g74fw" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47800b8f-a8ff-4f7e-b7b2-0b0723bd00f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47800b8f-a8ff-4f7e-b7b2-0b0723bd00f5",
      "name": "47800b8f-a8ff-4f7e-b7b2-0b0723bd00f5",
      "status": "Failed",
      "startTime": "2026-08-21T23:45:19.683338639Z",
      "endTime": "2026-08-21T23:45:20.631024642Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-215447 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-7jwrlb" to 4.23.0-0.nightly-multi-2026-08-21-150158
Unexpected error:
    <*fmt.wrapError | 0xc00071fec0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-7jwrlb" in resourcegroup="rg-cp-ystream-upgrade-4-23-ddtbb45g74fw" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47800b8f-a8ff-4f7e-b7b2-0b0723bd00f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/47800b8f-a8ff-4f7e-b7b2-0b0723bd00f5",
      "name": "47800b8f-a8ff-4f7e-b7b2-0b0723bd00f5",
      "status": "Failed",
      "startTime": "2026-08-21T23:45:19.683338639Z",
      "endTime": "2026-08-21T23:45:20.631024642Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-20-215447 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090937171266506752
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-21-221014 to 4.23.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-21-221014 to 4.23.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-z5pqzx" to 4.23.0-0.nightly-multi-2026-08-22-063717
Unexpected error:
    <*fmt.wrapError | 0xc0012ac660>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-z5pqzx" in resourcegroup="rg-cp-ystream-upgrade-4-23-227n7tlwrhnd" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef2c5e8e-3ad7-4179-842e-7126874115be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef2c5e8e-3ad7-4179-842e-7126874115be",
      "name": "ef2c5e8e-3ad7-4179-842e-7126874115be",
      "status": "Failed",
      "startTime": "2026-08-22T23:47:18.723854025Z",
      "endTime": "2026-08-22T23:47:19.754634857Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-21-221014 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-z5pqzx" to 4.23.0-0.nightly-multi-2026-08-22-063717
Unexpected error:
    <*fmt.wrapError | 0xc0012ac660>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-z5pqzx" in resourcegroup="rg-cp-ystream-upgrade-4-23-227n7tlwrhnd" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef2c5e8e-3ad7-4179-842e-7126874115be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef2c5e8e-3ad7-4179-842e-7126874115be",
      "name": "ef2c5e8e-3ad7-4179-842e-7126874115be",
      "status": "Failed",
      "startTime": "2026-08-22T23:47:18.723854025Z",
      "endTime": "2026-08-22T23:47:19.754634857Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-21-221014 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2091299562504327168
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-24-152026 to 5.1.0: no reachable ...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-24-152026 to 5.1.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-q6hwb9" to 5.1.0-0.nightly-multi-2026-08-24-111752
Unexpected error:
    <*fmt.wrapError | 0xc000a25da0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-q6hwb9" in resourcegroup="rg-cp-ystream-upgrade-5-1-kgxwzfcjcc2b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e23a315-fb36-4d2d-bc8a-8cee31204e59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e23a315-fb36-4d2d-bc8a-8cee31204e59",
      "name": "7e23a315-fb36-4d2d-bc8a-8cee31204e59",
      "status": "Failed",
      "startTime": "2026-08-24T23:51:24.1449981Z",
      "endTime": "2026-08-24T23:51:30.86207539Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-24-152026 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:217]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-q6hwb9" to 5.1.0-0.nightly-multi-2026-08-24-111752
Unexpected error:
    <*fmt.wrapError | 0xc000a25da0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-1-q6hwb9" in resourcegroup="rg-cp-ystream-upgrade-5-1-kgxwzfcjcc2b" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e23a315-fb36-4d2d-bc8a-8cee31204e59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e23a315-fb36-4d2d-bc8a-8cee31204e59",
      "name": "7e23a315-fb36-4d2d-bc8a-8cee31204e59",
      "status": "Failed",
      "startTime": "2026-08-24T23:51:24.1449981Z",
      "endTime": "2026-08-24T23:51:30.86207539Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 5.0.0-0.nightly-multi-2026-08-24-152026 to 5.1.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2092024382393159680
VerifyVMOSDiskCustomerEncryption(nodePool=<nodepool>) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-z8ktk-l4d4d_OSDisk for VM des-encrypt-des...
full failure pattern:
VerifyVMOSDiskCustomerEncryption(nodePool=<nodepool>) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-z8ktk-l4d4d_OSDisk for VM des-encrypt-des-np-z8ktk-l4d4d has encryption type EncryptionAtRestWithPlatformKey, expected
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:163]: cluster verification failed
Unexpected error:
    <*errors.joinError | 0xc0013905a0>: 
    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-z8ktk-l4d4d_OSDisk for VM des-encrypt-des-np-z8ktk-l4d4d has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey
    disk des-encrypt-des-np-z8ktk-t2cjq_OSDisk for VM des-encrypt-des-np-z8ktk-t2cjq has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:163]: cluster verification failed
Unexpected error:
    <*errors.joinError | 0xc0013905a0>: 
    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: OS disk encryption verification failed for 2/2 VMs: disk des-encrypt-des-np-z8ktk-l4d4d_OSDisk for VM des-encrypt-des-np-z8ktk-l4d4d has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey
    disk des-encrypt-des-np-z8ktk-t2cjq_OSDisk for VM des-encrypt-des-np-z8ktk-t2cjq has encryption type EncryptionAtRestWithPlatformKey, expected EncryptionAtRestWithCustomerKey
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089532925845442560
failed to update cluster "<cluster>" authorized CIDRs to include test runner IP
full failure pattern:
failed to update cluster "<cluster>" authorized CIDRs to include test runner IP
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 1 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:249]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:249]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2091299562504327168
x509: certificate signed by unknown authority
full failure pattern:
x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.45%1 of 29 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0INT, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090024653178277888

Environment: STG

Failure patterns: 54 · Failures matched: 261 · Total runs: 26 · Runs affected: 24

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days
e2e934.62%9 of 26 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 7 daysShow trend detailsAug 17: 0 · Aug 18: 3 · Aug 19: 2 · Aug 20: 2 · Aug 21: 1 · Aug 22: 1 · Aug 23: 1 · Aug 24: 1DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc002ac4480>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gp6bzmxwxfps, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-gp6bzmxwxfps" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gp6bzmxwxfps, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc002ac4480>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gp6bzmxwxfps, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-gp6bzmxwxfps" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gp6bzmxwxfps, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc001a9cb70>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-4mnnfzb2nhtb, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-4mnnfzb2nhtb" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-4mnnfzb2nhtb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc001a9cb70>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-4mnnfzb2nhtb, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-4mnnfzb2nhtb" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-4mnnfzb2nhtb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc002b9e330>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-fg554p8tjmmf, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fg554p8tjmmf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-fg554p8tjmmf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc002b9e330>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-fg554p8tjmmf, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fg554p8tjmmf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-fg554p8tjmmf, error: context deadline exceeded
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public7
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.12
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
Affected runs (9)
Date (UTC)Associated PRProw job
n/a2092024382493822976
n/a2091661989980934144
n/a2091299562508521472
n/a2090937171316838400
n/a2090574634805104640
n/a2090257540762832896
n/a2090212361167179776
n/a2089849852383989760
n/a2089532925895774208
tls: failed to verify certificate: x509: certificate signed by unknown authority
full failure pattern:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e830.77%8 of 26 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 2 · Aug 21: 10 · Aug 22: 17 · Aug 23: 18 · Aug 24: 8INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:196]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001990018>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-58vq7.apps.aro.cilium-cluster.xjbu.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:196]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001990018>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-58vq7.apps.aro.cilium-cluster.xjbu.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:336]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0023089a8>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-gd2m6.apps.aro.agg-cluster.sh9x.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:336]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0023089a8>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-gd2m6.apps.aro.agg-cluster.sh9x.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002bc60a8>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-2xkbq.apps.aro.cilium-cl.yqg3.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:175]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002bc60a8>: 
    VerifySimpleWebApp failed: [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-2xkbq.apps.aro.cilium-cl.yqg3.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
Contributing tests (19)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.213
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.223
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.203
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.213
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.223
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.233
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.03
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.13
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a HCP cluster and use cilium CNI plugin3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (8)
Date (UTC)Associated PRProw job
n/a2091707289328160768
n/a2091661989980934144
n/a2091344896731385856
n/a2091299562508521472
n/a2090982470185914368
n/a2090937171316838400
n/a2090574634805104640
n/a2090419881110409216
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e519.23%5 of 26 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsAug 17: 1 · Aug 18: 1 · Aug 19: 5 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001c244b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-v62dhvmn5xjx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/433c7b9f-9ee6-405a-ac4b-61c2aba8b5b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/433c7b9f-9ee6-405a-ac4b-61c2aba8b5b8",
      "name": "433c7b9f-9ee6-405a-ac4b-61c2aba8b5b8",
      "status": "Failed",
      "startTime": "2026-08-19T23:23:38.105024916Z",
      "endTime": "2026-08-19T23:47:40.969477415Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ugi7ah3g0hdhfnrdfe02s4641un92 still exists (deletion dispatched at 2026-08-19T23:23:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001c244b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-v62dhvmn5xjx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/433c7b9f-9ee6-405a-ac4b-61c2aba8b5b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/433c7b9f-9ee6-405a-ac4b-61c2aba8b5b8",
      "name": "433c7b9f-9ee6-405a-ac4b-61c2aba8b5b8",
      "status": "Failed",
      "startTime": "2026-08-19T23:23:38.105024916Z",
      "endTime": "2026-08-19T23:47:40.969477415Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ugi7ah3g0hdhfnrdfe02s4641un92 still exists (deletion dispatched at 2026-08-19T23:23:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0013d62d0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-zk8j9gsrf47s" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83d822b3-ac2e-4cc4-966c-eab4ee8e8602
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83d822b3-ac2e-4cc4-966c-eab4ee8e8602",
      "name": "83d822b3-ac2e-4cc4-966c-eab4ee8e8602",
      "status": "Failed",
      "startTime": "2026-08-19T13:56:53.166820811Z",
      "endTime": "2026-08-19T14:20:55.153951347Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9m491m1ahtrh178oqepopioqrgcdms still exists (deletion dispatched at 2026-08-19T13:56:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0013d62d0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-zk8j9gsrf47s" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83d822b3-ac2e-4cc4-966c-eab4ee8e8602
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/83d822b3-ac2e-4cc4-966c-eab4ee8e8602",
      "name": "83d822b3-ac2e-4cc4-966c-eab4ee8e8602",
      "status": "Failed",
      "startTime": "2026-08-19T13:56:53.166820811Z",
      "endTime": "2026-08-19T14:20:55.153951347Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9m491m1ahtrh178oqepopioqrgcdms still exists (deletion dispatched at 2026-08-19T13:56:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00112a408>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-rns7wq" in resourcegroup="rg-zstream-upgrade-4-20-vhxh5b6zszlg" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1928d5f-397e-4b68-89ec-79ef6c7cd4ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1928d5f-397e-4b68-89ec-79ef6c7cd4ac",
      "name": "e1928d5f-397e-4b68-89ec-79ef6c7cd4ac",
      "status": "Failed",
      "startTime": "2026-08-19T14:13:37.892189859Z",
      "endTime": "2026-08-19T14:37:44.990617474Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9mivqf2eovoc0ctilj3pgriaobnqh1 still exists (deletion dispatched at 2026-08-19T14:13:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00112a408>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-rns7wq" in resourcegroup="rg-zstream-upgrade-4-20-vhxh5b6zszlg" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1928d5f-397e-4b68-89ec-79ef6c7cd4ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e1928d5f-397e-4b68-89ec-79ef6c7cd4ac",
      "name": "e1928d5f-397e-4b68-89ec-79ef6c7cd4ac",
      "status": "Failed",
      "startTime": "2026-08-19T14:13:37.892189859Z",
      "endTime": "2026-08-19T14:37:44.990617474Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9mivqf2eovoc0ctilj3pgriaobnqh1 still exists (deletion dispatched at 2026-08-19T14:13:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to update node pool labels and taints1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2090212361167179776
n/a2090064577516539904
n/a2089951420068401152
n/a2089895148551933952
n/a2089849852383989760
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e519.23%5 of 26 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 3 · Aug 20: 2 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00118e480>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-h759ch" in resourcegroup="rg-nightly-4-20-mzr4r5lz42qp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55ed743f-d106-4adb-b590-caa3369caeb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55ed743f-d106-4adb-b590-caa3369caeb3",
      "name": "55ed743f-d106-4adb-b590-caa3369caeb3",
      "status": "Failed",
      "startTime": "2026-08-24T23:25:48.884097042Z",
      "endTime": "2026-08-24T23:49:50.709200875Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd7vm7aq4os1ksjghr3o1msj3ksno6l still exists (deletion dispatched at 2026-08-24T23:25:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00118e480>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-h759ch" in resourcegroup="rg-nightly-4-20-mzr4r5lz42qp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55ed743f-d106-4adb-b590-caa3369caeb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/55ed743f-d106-4adb-b590-caa3369caeb3",
      "name": "55ed743f-d106-4adb-b590-caa3369caeb3",
      "status": "Failed",
      "startTime": "2026-08-24T23:25:48.884097042Z",
      "endTime": "2026-08-24T23:49:50.709200875Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sd7vm7aq4os1ksjghr3o1msj3ksno6l still exists (deletion dispatched at 2026-08-24T23:25:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00111e108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-95b2mn6x5nc7" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425",
      "name": "56a5601c-84db-4c4c-afe3-fa090d79a425",
      "status": "Failed",
      "startTime": "2026-08-20T09:48:30.429013476Z",
      "endTime": "2026-08-20T10:12:32.646924727Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7kb8vek727d6pbio7effbv09ei0ua still exists (deletion dispatched at 2026-08-20T09:48:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00111e108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-95b2mn6x5nc7" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/56a5601c-84db-4c4c-afe3-fa090d79a425",
      "name": "56a5601c-84db-4c4c-afe3-fa090d79a425",
      "status": "Failed",
      "startTime": "2026-08-20T09:48:30.429013476Z",
      "endTime": "2026-08-20T10:12:32.646924727Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7kb8vek727d6pbio7effbv09ei0ua still exists (deletion dispatched at 2026-08-20T09:48:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc002d34258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-x729kk" in resourcegroup="rg-candidate-4-22-6wvs8lznc8fp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c",
      "name": "bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c",
      "status": "Failed",
      "startTime": "2026-08-20T09:47:07.957787635Z",
      "endTime": "2026-08-20T10:11:12.923142399Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7h5aduc2uvihvfe2vdijl20h32glr still exists (deletion dispatched at 2026-08-20T09:47:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc002d34258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-x729kk" in resourcegroup="rg-candidate-4-22-6wvs8lznc8fp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c",
      "name": "bdc947fc-d7e9-4fa7-ab9f-3f9b996c730c",
      "status": "Failed",
      "startTime": "2026-08-20T09:47:07.957787635Z",
      "endTime": "2026-08-20T10:11:12.923142399Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sa7h5aduc2uvihvfe2vdijl20h32glr still exists (deletion dispatched at 2026-08-20T09:47:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2092024382493822976
n/a2090356957637513216
n/a2090212361167179776
n/a2090159303129305088
n/a2089990802548723712
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
e2e415.38%4 of 26 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsAug 17: 2 · Aug 18: 0 · Aug 19: 0 · Aug 20: 4 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000e8ea40>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-lf6jg45wtwxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62db7451-86ec-4c6a-9227-bc775f891ebb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62db7451-86ec-4c6a-9227-bc775f891ebb",
      "name": "62db7451-86ec-4c6a-9227-bc775f891ebb",
      "status": "Failed",
      "startTime": "2026-08-24T23:05:01.131009318Z",
      "endTime": "2026-08-24T23:24:05.637766062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000e8ea40>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-lf6jg45wtwxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62db7451-86ec-4c6a-9227-bc775f891ebb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62db7451-86ec-4c6a-9227-bc775f891ebb",
      "name": "62db7451-86ec-4c6a-9227-bc775f891ebb",
      "status": "Failed",
      "startTime": "2026-08-24T23:05:01.131009318Z",
      "endTime": "2026-08-24T23:24:05.637766062Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000f3f2e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sh428kn5d89n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3af2b0d-5fad-4125-9764-31b214f8fee1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3af2b0d-5fad-4125-9764-31b214f8fee1",
      "name": "b3af2b0d-5fad-4125-9764-31b214f8fee1",
      "status": "Failed",
      "startTime": "2026-08-21T23:04:23.850321515Z",
      "endTime": "2026-08-21T23:23:28.45210372Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000f3f2e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sh428kn5d89n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3af2b0d-5fad-4125-9764-31b214f8fee1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b3af2b0d-5fad-4125-9764-31b214f8fee1",
      "name": "b3af2b0d-5fad-4125-9764-31b214f8fee1",
      "status": "Failed",
      "startTime": "2026-08-21T23:04:23.850321515Z",
      "endTime": "2026-08-21T23:23:28.45210372Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-pt5scqn6n82h/cluster-candidate-4-21-28flhq should provision
Unexpected error:
    <*fmt.wrapError | 0xc00124eb60>: 
    failed to create HCP cluster cluster-candidate-4-21-28flhq: failed waiting for cluster="cluster-candidate-4-21-28flhq" in resourcegroup="rg-candidate-4-21-pt5scqn6n82h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc",
      "name": "332f74a8-d0dc-4e4a-9737-70119dac90bc",
      "status": "Failed",
      "startTime": "2026-08-20T20:53:42.523371474Z",
      "endTime": "2026-08-20T21:12:49.332578125Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-pt5scqn6n82h/cluster-candidate-4-21-28flhq should provision
Unexpected error:
    <*fmt.wrapError | 0xc00124eb60>: 
    failed to create HCP cluster cluster-candidate-4-21-28flhq: failed waiting for cluster="cluster-candidate-4-21-28flhq" in resourcegroup="rg-candidate-4-21-pt5scqn6n82h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/332f74a8-d0dc-4e4a-9737-70119dac90bc",
      "name": "332f74a8-d0dc-4e4a-9737-70119dac90bc",
      "status": "Failed",
      "startTime": "2026-08-20T20:53:42.523371474Z",
      "endTime": "2026-08-20T21:12:49.332578125Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2092024382493822976
n/a2090937171316838400
n/a2090538506672672768
n/a2090356957637513216
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e415.38%4 of 26 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsAug 17: 4 · Aug 18: 1 · Aug 19: 0 · Aug 20: 2 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:139]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc0006ae100>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-s82wq8qv974g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/389f219e-5f8d-4b6a-9c72-fdd66be0b32d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/389f219e-5f8d-4b6a-9c72-fdd66be0b32d",
      "name": "389f219e-5f8d-4b6a-9c72-fdd66be0b32d",
      "status": "Failed",
      "startTime": "2026-08-21T23:17:03.253121618Z",
      "endTime": "2026-08-21T23:36:09.200090899Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:139]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc0006ae100>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-s82wq8qv974g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/389f219e-5f8d-4b6a-9c72-fdd66be0b32d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/389f219e-5f8d-4b6a-9c72-fdd66be0b32d",
      "name": "389f219e-5f8d-4b6a-9c72-fdd66be0b32d",
      "status": "Failed",
      "startTime": "2026-08-21T23:17:03.253121618Z",
      "endTime": "2026-08-21T23:36:09.200090899Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-21 with version 4.21.28
Unexpected error:
    <*fmt.wrapError | 0xc0010bc7e0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-8mw8p7" in resourcegroup="rg-np-version-upgrade-8mw8p7-jzxbk7cd5ktl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d",
      "name": "217c5b5b-876e-4dfd-aa27-a2f787a06b7d",
      "status": "Failed",
      "startTime": "2026-08-20T09:52:42.575458861Z",
      "endTime": "2026-08-20T10:11:42.477430185Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:166]: failed to create node pool npupgrade-4-21 with version 4.21.28
Unexpected error:
    <*fmt.wrapError | 0xc0010bc7e0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-8mw8p7" in resourcegroup="rg-np-version-upgrade-8mw8p7-jzxbk7cd5ktl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/217c5b5b-876e-4dfd-aa27-a2f787a06b7d",
      "name": "217c5b5b-876e-4dfd-aa27-a2f787a06b7d",
      "status": "Failed",
      "startTime": "2026-08-20T09:52:42.575458861Z",
      "endTime": "2026-08-20T10:11:42.477430185Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc0009eaaa0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qjg88gwvq5xj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4",
      "name": "c2facd99-ace4-452a-af16-9d4629a551d4",
      "status": "Failed",
      "startTime": "2026-08-20T04:35:11.139884226Z",
      "endTime": "2026-08-20T04:54:12.658497794Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc0009eaaa0>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qjg88gwvq5xj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c2facd99-ace4-452a-af16-9d4629a551d4",
      "name": "c2facd99-ace4-452a-af16-9d4629a551d4",
      "status": "Failed",
      "startTime": "2026-08-20T04:35:11.139884226Z",
      "endTime": "2026-08-20T04:54:12.658497794Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2090937171316838400
n/a2090356957637513216
n/a2090276754840096768
n/a2089645995238363136
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e27.69%2 of 26 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsAug 17: 2 · Aug 18: 1 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.005s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.006s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 1980.006s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090619934307520512
n/a2089532925895774208
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e830.77%8 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 4 · Aug 20: 4 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011964b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-h8fb74" in resourcegroup="rg-np-version-upgrade-h8fb74-s5mqnw56gx4w" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/701376c3-ab04-47a8-a1ef-8375724f8f86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/701376c3-ab04-47a8-a1ef-8375724f8f86",
      "name": "701376c3-ab04-47a8-a1ef-8375724f8f86",
      "status": "Failed",
      "startTime": "2026-08-22T00:26:59.095965518Z",
      "endTime": "2026-08-22T00:51:01.463363445Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb99fkh6uh3oo5nr4qni7jhed5o0tgt still exists (deletion dispatched at 2026-08-22T00:26:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011964b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-h8fb74" in resourcegroup="rg-np-version-upgrade-h8fb74-s5mqnw56gx4w" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/701376c3-ab04-47a8-a1ef-8375724f8f86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/701376c3-ab04-47a8-a1ef-8375724f8f86",
      "name": "701376c3-ab04-47a8-a1ef-8375724f8f86",
      "status": "Failed",
      "startTime": "2026-08-22T00:26:59.095965518Z",
      "endTime": "2026-08-22T00:51:01.463363445Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sb99fkh6uh3oo5nr4qni7jhed5o0tgt still exists (deletion dispatched at 2026-08-22T00:26:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0013c8138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2p9lgg975wsf" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5710f93-c135-4ecd-84ba-bd766a4f3157
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5710f93-c135-4ecd-84ba-bd766a4f3157",
      "name": "d5710f93-c135-4ecd-84ba-bd766a4f3157",
      "status": "Failed",
      "startTime": "2026-08-21T02:51:19.851300811Z",
      "endTime": "2026-08-21T03:15:24.947768689Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2samnf949m9mks8utf8hvjl3i9ocmvd0 still exists (deletion dispatched at 2026-08-21T02:51:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0013c8138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2p9lgg975wsf" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5710f93-c135-4ecd-84ba-bd766a4f3157
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d5710f93-c135-4ecd-84ba-bd766a4f3157",
      "name": "d5710f93-c135-4ecd-84ba-bd766a4f3157",
      "status": "Failed",
      "startTime": "2026-08-21T02:51:19.851300811Z",
      "endTime": "2026-08-21T03:15:24.947768689Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2samnf949m9mks8utf8hvjl3i9ocmvd0 still exists (deletion dispatched at 2026-08-21T02:51:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011f24f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listwvkfxb" in resourcegroup="ea-list-q8zktqx9k8tx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050",
      "name": "7e8f24bc-6934-4503-b388-7fea73015050",
      "status": "Failed",
      "startTime": "2026-08-20T14:04:49.548371908Z",
      "endTime": "2026-08-20T14:28:52.367967433Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sabeurund3r70lf12nj56m098p5j7j9 still exists (deletion dispatched at 2026-08-20T14:04:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011f24f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listwvkfxb" in resourcegroup="ea-list-q8zktqx9k8tx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7e8f24bc-6934-4503-b388-7fea73015050",
      "name": "7e8f24bc-6934-4503-b388-7fea73015050",
      "status": "Failed",
      "startTime": "2026-08-20T14:04:49.548371908Z",
      "endTime": "2026-08-20T14:28:52.367967433Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sabeurund3r70lf12nj56m098p5j7j9 still exists (deletion dispatched at 2026-08-20T14:04:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (8)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (8)
Date (UTC)Associated PRProw job
n/a2090937171316838400
n/a2090619934307520512
n/a2090419881110409216
n/a2090356957637513216
n/a2090257540762832896
n/a2090159303129305088
n/a2090064577516539904
n/a2089951420068401152
x509: certificate signed by unknown authority
full failure pattern:
x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e726.92%7 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 1 · Aug 22: 2 · Aug 23: 2 · Aug 24: 1INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.001s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.001s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.000s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:236]: Timed out after 600.000s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create an HCP cluster and validate TLS certificates3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2091707289328160768
n/a2091661989980934144
n/a2091344896731385856
n/a2091299562508521472
n/a2090982470185914368
n/a2090937171316838400
n/a2090356957637513216
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e519.23%5 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 3 · Aug 20: 0 · Aug 21: 4 · Aug 22: 4 · Aug 23: 4 · Aug 24: 4DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0012c61b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22.0-0.nightly-multi-2026-08-23-080550, targetMinor=4.23.0-0.nightly-multi-2026-08-24-110923) failed: clusterversion status.history has no version in target minor "4.23.0-0.nightly-multi-2026-08-24-110923"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0012c61b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22.0-0.nightly-multi-2026-08-23-080550, targetMinor=4.23.0-0.nightly-multi-2026-08-24-110923) failed: clusterversion status.history has no version in target minor "4.23.0-0.nightly-multi-2026-08-24-110923"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc000013c68>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-24-135049, targetMinor=4.21.0-0.nightly-multi-2026-08-24-143120) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-24-143120"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc000013c68>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-24-135049, targetMinor=4.21.0-0.nightly-multi-2026-08-24-143120) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-24-143120"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc001423c38>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-24-143120, targetMinor=4.22.0-0.nightly-multi-2026-08-23-080550) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-23-080550"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:226]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.joinError | 0xc001423c38>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-08-24-143120, targetMinor=4.22.0-0.nightly-multi-2026-08-23-080550) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-08-23-080550"
    ...
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor4
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2092024382493822976
n/a2091661989980934144
n/a2091299562508521472
n/a2090937171316838400
n/a2090212361167179776
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
full failure pattern:
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e519.23%5 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 2 · Aug 23: 2 · Aug 24: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:156]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc0017be510>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:156]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc0017be510>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:156]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001391020>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:156]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001391020>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:156]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc0013e8f30>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:156]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc0013e8f30>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal2
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2091707289328160768
n/a2091661989980934144
n/a2091344896731385856
n/a2091299562508521472
n/a2090982470185914368
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e415.38%4 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 3 · Aug 19: 2 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001c54180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-8hvpj6vtkqlf" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d19e078c-f7b8-4be7-ba0d-0921f4e3a585
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d19e078c-f7b8-4be7-ba0d-0921f4e3a585",
      "name": "d19e078c-f7b8-4be7-ba0d-0921f4e3a585",
      "status": "Failed",
      "startTime": "2026-08-19T23:47:16.516011942Z",
      "endTime": "2026-08-20T00:11:22.445943521Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9uhn33u6bvucvlivpurd7qbjud4jvg still exists (deletion dispatched at 2026-08-19T23:47:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001c54180>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-8hvpj6vtkqlf" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d19e078c-f7b8-4be7-ba0d-0921f4e3a585
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d19e078c-f7b8-4be7-ba0d-0921f4e3a585",
      "name": "d19e078c-f7b8-4be7-ba0d-0921f4e3a585",
      "status": "Failed",
      "startTime": "2026-08-19T23:47:16.516011942Z",
      "endTime": "2026-08-20T00:11:22.445943521Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9uhn33u6bvucvlivpurd7qbjud4jvg still exists (deletion dispatched at 2026-08-19T23:47:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0017d6138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-qxfhsjtcs5wr" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c2f0336-078f-4209-9e4d-a24e8e832089
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c2f0336-078f-4209-9e4d-a24e8e832089",
      "name": "2c2f0336-078f-4209-9e4d-a24e8e832089",
      "status": "Failed",
      "startTime": "2026-08-19T02:31:15.704278812Z",
      "endTime": "2026-08-19T02:55:20.050088368Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9c23lo4u800am915f4oldars1ighvd still exists (deletion dispatched at 2026-08-19T02:31:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0017d6138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-ingress" in resourcegroup="private-ingress-qxfhsjtcs5wr" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c2f0336-078f-4209-9e4d-a24e8e832089
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c2f0336-078f-4209-9e4d-a24e8e832089",
      "name": "2c2f0336-078f-4209-9e4d-a24e8e832089",
      "status": "Failed",
      "startTime": "2026-08-19T02:31:15.704278812Z",
      "endTime": "2026-08-19T02:55:20.050088368Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9c23lo4u800am915f4oldars1ighvd still exists (deletion dispatched at 2026-08-19T02:31:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001aa82b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fr2tsbrfr7wk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e32155a0-2078-4425-ba3d-ae626da965da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e32155a0-2078-4425-ba3d-ae626da965da",
      "name": "e32155a0-2078-4425-ba3d-ae626da965da",
      "status": "Failed",
      "startTime": "2026-08-19T00:50:14.353170498Z",
      "endTime": "2026-08-19T01:14:23.20531535Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9antrh42pdgmli6evu71asbu6o4q65 still exists (deletion dispatched at 2026-08-19T00:50:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001aa82b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fr2tsbrfr7wk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e32155a0-2078-4425-ba3d-ae626da965da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e32155a0-2078-4425-ba3d-ae626da965da",
      "name": "e32155a0-2078-4425-ba3d-ae626da965da",
      "status": "Failed",
      "startTime": "2026-08-19T00:50:14.353170498Z",
      "endTime": "2026-08-19T01:14:23.20531535Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9antrh42pdgmli6evu71asbu6o4q65 still exists (deletion dispatched at 2026-08-19T00:50:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2090212361167179776
n/a2089895148551933952
n/a2089849852383989760
n/a2089760421908058112
node pool <nodepool> verification failed: <node> (no version in containerRuntimeVersion "cri-o://1
full failure pattern:
node pool <nodepool> verification failed: <node> (no version in containerRuntimeVersion "cri-o://1
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e415.38%4 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 1 · Aug 23: 1 · Aug 24: 1PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.errorString | 0xc0012c0d10>: 
    node pool upgrade verification failed: w2k5s8r8i0t4x2o-nps-420-422-bqdkd-4jngb (no version in containerRuntimeVersion "cri-o://1.35.4"); w2k5s8r8i0t4x2o-nps-420-422-bqdkd-wx6fr (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"w2k5s8r8i0t4x2o-nps-420-422-bqdkd-4jngb","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380"]},{"name":"w2k5s8r8i0t4x2o-nps-420-422-bqdkd-wx6fr","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.errorString | 0xc0012c0d10>: 
    node pool upgrade verification failed: w2k5s8r8i0t4x2o-nps-420-422-bqdkd-4jngb (no version in containerRuntimeVersion "cri-o://1.35.4"); w2k5s8r8i0t4x2o-nps-420-422-bqdkd-wx6fr (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"w2k5s8r8i0t4x2o-nps-420-422-bqdkd-4jngb","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380"]},{"name":"w2k5s8r8i0t4x2o-nps-420-422-bqdkd-wx6fr","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc002a14f40>: 
    node pool upgrade verification failed: x2t8p7p8m2s5y4s-nps-420-422-mqm7r-2pww5 (no version in containerRuntimeVersion "cri-o://1.35.4"); x2t8p7p8m2s5y4s-nps-420-422-mqm7r-vvlrc (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"x2t8p7p8m2s5y4s-nps-420-422-mqm7r-2pww5","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"x2t8p7p8m2s5y4s-nps-420-422-mqm7r-vvlrc","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc002a14f40>: 
    node pool upgrade verification failed: x2t8p7p8m2s5y4s-nps-420-422-mqm7r-2pww5 (no version in containerRuntimeVersion "cri-o://1.35.4"); x2t8p7p8m2s5y4s-nps-420-422-mqm7r-vvlrc (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"x2t8p7p8m2s5y4s-nps-420-422-mqm7r-2pww5","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"x2t8p7p8m2s5y4s-nps-420-422-mqm7r-vvlrc","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6485404900b5004ca3a7e5aaa77e5c6b5bb7240121161d078bc15e9c3ef5162","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3228f0dcb7a11d8eb20bb3c8b15da2d77a3fa8507223aae0c5777842a714728","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:458ccf68c0287122e5d22482719714b01b4ed8396787e1094786ec98449a2df0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82c7829e1f6029dce1bef4743ca42327413d36214d9c40b09f8541dc68f2afad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9bf72765f0b6aba23570c1c09399e99a3c86a5160742943afd4d05e3db4bf9e7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7682214f2449957bd850f65d766bb182aced114511180df250066c89459018c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.errorString | 0xc002bd0580>: 
    node pool upgrade verification failed: g2a9g8l9y5u0b6x-nps-420-422-96vnb-jdgtr (no version in containerRuntimeVersion "cri-o://1.35.4"); g2a9g8l9y5u0b6x-nps-420-422-96vnb-tl58m (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"g2a9g8l9y5u0b6x-nps-420-422-96vnb-jdgtr","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"g2a9g8l9y5u0b6x-nps-420-422-96vnb-tl58m","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.errorString | 0xc002bd0580>: 
    node pool upgrade verification failed: g2a9g8l9y5u0b6x-nps-420-422-96vnb-jdgtr (no version in containerRuntimeVersion "cri-o://1.35.4"); g2a9g8l9y5u0b6x-nps-420-422-96vnb-tl58m (no version in containerRuntimeVersion "cri-o://1.35.4"); nodes=[{"name":"g2a9g8l9y5u0b6x-nps-420-422-96vnb-jdgtr","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20d78771136ba2e9a7aa535296eafe94ef196045f3a00a04a7f08d3fa6ad2d9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6810e13d061f10e267e73c3fde34d10a89644114a034103cadaa4f5ad751571","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cfa918ae07c4106e9cb91235788e607a140c3de42f02a0f81614092c0f5fb93","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71083ff15d6aa4658fd679d03a905735a1a54fc49de0d72eab4e3366e498d41c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a605bd3607ba38f9904b3246eb04e657dadccce5359f5b76af74e0a72bb87cb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd2343f595006f558a5b149f519d84bb3375a3103ff1ed4037830c502209d9d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55bb6fc2dd45c4caaea16fb1f008eeb54f4b3b7a9ac852bcac140b61dd9dc85f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afd2d3a5b70a05d98507e79c997593dcf9db357f16d3355a4868e632b5292719","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1d85150fcd72f4d601c093b5b2ba779da89c9ead8d4dd53b596c8b89d16ccc08","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8485f1881ece91e286057351277ac800f345a22e4d1bc1ab3ad7e73de17cef6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144e0b078a15bb7fd714e4cf53069650721ff9fe2b73fd05921a961696721afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4292941aed2127039306505411bc6abf798085d7d15c0089231fec79671d624b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:acabe3909ec92663ee0980963d10212c3e08e9f165eca2700f146aec32c7d1fa","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9ba2395e6a8fd2b6c5afc4a6781a31c05f72b5679f77a3cf6ebc7556048e47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]},{"name":"g2a9g8l9y5u0b6x-nps-420-422-96vnb-tl58m","ready":true,"containerRuntimeVersion":"cri-o://1.35.4","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c2a780cb30d9032f2f6b40626e575dd48ecbd5808557398bf6c7baefda26ccc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e69bac9d7618698a57aa3169948f353f3abe50ba67c5f041841ba4443657a5f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:617996348f24e8651faa289768d9d01f2f855de9a9793b81d02c5cefc92ef1a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:856f008072cf2e9c50841f69b0dfba1703ab86a76b7a850290c5fc747fa57214","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0acb9aa82d2c0eb4394609dca6da3a950226fc73f10bd38df3bf2adfb053edf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a693b8a10280fb9dec32eec3845cadb993134ee38afc2cb2890f87f0dc99111","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:270b2f636233a862bb97746e3b97c69320b499f53590b91ea9f6ca9798e6b95b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29cd209b806f92660233e4b67f648ede22ddc20cba2c6a6bcbda992022830c52","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b38493011dd1f654b1667e9c131cab5a11344a2ac32f414d457c0444559f12d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bfa95dcd4a5aa349ba3fc73a0d653779cb753c344a53273cf05b9c32c7055da1","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3ff0e77084bf0bd36460aa2532fe4a9ff67e0a50fdad3fa844646c2a956b476","arohcpocpstg.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7bc458544a67245f4465b1de78924f32c827e4e53ab9a17cfa72f5be1f1a0ec","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:242c82f60fcf35b3c09f2de489d143ac4a1418bf6ab70f0b6d0959045c87604c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfcb5d21731e8654ae9709ce20b0f1ad628b9a999cb72df8cc6142076cf6b22a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ca67f742652a990f253f006c1503a8b6bf1684b22b7a2dd9b8339efa0bc42f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3d4cc8de6d4b03dde4c806a8ada844bbbfb03f47b2f6bfc923d24f8c79efa57","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39035c7e2de8a140db90ec620f9f7d8d6b1ae9a5a4eae6238b02526aba4b44a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6206df7088a9ce0b19839198d49b75de875422f9b872c770013237cff1c1babd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee6f3a63882ba5c2f7e761d9cdce292b55d0eb5ad199323d0ef1a07ddb08b08f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc17e5a66f94564111e8ee4913d23ac45b0a034c61a5debe9fbce732f5677732","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33ed2805042607c08b54e3040692fdb12763eaecaab721109c127e9a6e1e5648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b94009944719ea47e9ccb553ceee7c74a5983121c8658199012a6c0a09f3d147","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35a7e61db3b3f21a10de8ad2c4df0f52212e29dc8839a50369efb90687f073f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a30cd38d6591bbb0f81dc45b9face049a4fafa6bea60b020fb87cbcedfc8d060","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143b03c645cbcc539b21d973918a3d4cabc1b5b97b285ab6bfa28ac87fdc7fe2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb2c4564a9c0cb1a53c181cdb8c232b739bdcadf41e98103537b27809bb2306a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b8975696bfa7211144dd4b3519ad13dbcbe8ff793286cf5735203666fa55b56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cc662715c4f83ee4a129ddb533117bfaa95bea6dee22dad57a556bc1346092a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69128068c1b2145cb4424aa98ad3d4affed345eaf13216c71f6ee73fc6c702f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:833efdc864a51200ee03e4f7857cb3f76c6ba8f82c8031a59518dd3b903891c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a31f34b179cb7c2420d1d7d703405e50777e1b4ee3deb714d22d9c379f76b64","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5682112322399b773223dc1ae7843b1eaafc4a3e0d136966c49eaa02e520d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ae8698744c915c26420510e9cc7939f22c85fd5647413606472d70025e7ba47","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8486fa29fdf815374bf127b860642bed575917c44a8d5cc3a82f1f14c3123e48","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39ea6ed4567969c3f1f69f2a24a196c7c877b5974d29c367154598774b674aa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4330e3f6e6a1b7ddb2bc7136226d12e4f5aa495f66f2c53da15c354044c47b41","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64e7c1c2c1ebc3e7fd08a0ee88340c514a05b9731ba2319ddf56fe1b4c06ffd9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c45d3e850d9046d717b4f681f219afaa97170e0c343d38f4f89c7d8696a07ef2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c6f42dbedc645545c4210d86a0e85d51593f17587a66d30d1f056af06a0a59c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9cc52f6689e49349206f51e9fc55f9fd5b5cb0c25b445e5724016205384443b9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e2086907b8cd808dbdf49df4d6ff757c20d40883514462bd127760c277d6259","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a093b6e3244495c45d8578af53693f36a40de3b53f84919d230d6fddf455f7c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a7161008f29e26910d7dda7e6c0c1886560317624c6879802975cc24261a6b97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f41d92c9bebe76fb811f271e648dde65a376496ce7eca1d125d01559835dfcb7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:01fa2753a35ec50f96664a523526680954ede835c5103e3ca1cdb0489c9d8f05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0df345275aa56086515f626638a554573328c22be5ff4ef1f74b1cc4d72d145f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4f0ce6dfeb2f2f82936f92a775108170e19f27680b313d1ca22934b53a2ff95a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cad9747010b5addc26808a1ca5ab8c0402da153b1aacc5dab65d55446f3815f8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:619cb3f003814da880a998260a4ffc7d1430c05acb15885e659cf9728039d13b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79a712f6151d6c501567d4bcd140605b7e002aacd018d33065197c6ec73cf04e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a7dcb90f64661e7e4d5bc63d9cfe4c1ebbdff4695926f13210fc7a1dabec1bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba0c482f1212af758f2d44a0d269d1045caff4f69905092bb621ce9f9a570dc7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:705dbb8f3d1207b33e04f5e2edb6167e517e50765e29558cea501b8c4bd5d355","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:768eb63513b08318e9c633094d60b1a0ecdedc93bfaf22592f4f9ac4e57eeacd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:392b0ef8c36398f9441d31f80e4b233a418cc77c9e3530d1052c3250f1933f55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a924e215fbd2e04c200b4bd9e2e7b9667160066ca9fe5731e1a2e100f30ed79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8c5e8a02e870bc7d4f63f9d441d972a1de4b77c816cda9252c13e6d604c3678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dbf8d9e6099a2ec5c27e70fc506279c744f9ea26fbe4a601d5dae168e174adf1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3818f08ef58520bdbdbd668b304fdefd4f3952e4789849577a1f61ec5a29888c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ab8442cdb4ca07861755f7f98d983e60577cfc9c159daede9c96393c34c45a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ecadce9bc3968cbdab409761e31fc3e9706b0983bc14c093252b18bf36580db","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:85cdf2b18e07f5cdd642a3d1552280637637e92761d23087c1d1d302545c6e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b2aeb5f663f8fef2b5de811d6159d00d166c5eba7d4251c98ea5b521ae07f00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bcb807b6cee3e50ab1dc96db618f84b9d3963dc76507b63fe64db9dde3db5fd2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0031c188acf8712b7f7bd38f97ffaa55c72043f7fa4548c5e71f669bf43e25bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90d853bebbc4d60706b6bffcc0f080a7a1caad9d21cd45f249a4be59ade53c98","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69b1c16b5a1c98cfbc5fd38d3bb19ac93e5084b9b26a3e1ae3cee2349727a794","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95efae9f45e4496d1451c96bcc0866412a971ca687530dccf81de35c40c1b31d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:038474ed33c60f474f40784c4c356c770fa1f854363444de831549bda3868fb6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5719d6a2898ab37f669911d025da8c3bc58d8395280d5e2e958407c30076897d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3dbfa79dcf3c51b06d821d56a1652d2c38f342c982305feba4f28df712088399","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e8f394246df3e18e3522fb9d5a660c5683a41d948bb28687061f237cc1d3a533","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ba932355c992fffc26d5422047521b6fcc613c78f675ec8ec37a69c6e20c98f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e85d556d9da5c3e0e855123fdd192da299ff6a19f53b86ba8e613515165fb519","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a86b8cb0138eb2cb1415ea1a2b6a30608b4957110629b0bf8dc8a4e846077bcf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e740fd5af3633855e8b36af5d5b0a0c6d1037ffe73030bd7558bfdc03428ab68","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1160c49464450d4f4c53de5f988656cea9f0c8729c7a042ed7d0f6377c1cd99d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6d4b8be8609b3debbe7c9efdccf84282581d94ae334e02f34945a10e86e70d68","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpstg.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1961efa7640ec88c399896e7a06e4f6f1c234e2d512eeb94c055980637d904e1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6c354101b4a3b9984b529150076bc671358c87a785fbf690a5c6dfab51173ae0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:08e92773fa95d4365a5e4e685742b2e1a10d4bb7e4f675f6a6e6e83b77a84f91","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31d154415a375b088aa2b3648b9c0fb0889aeb39089aa46395112a5306fcd212","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:310b2836a335755a8f429d3f2ab57c01d9cbd8b93912963d28d7a261534c16fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44143796439f39eeaa08c6e8d2067c6f78ceebe481154267f89c4e9a7e960380","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:072f527c571a27e80b8a398e8a571de6b56659b1f964f76728c345e4d0807ad7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36eabc5b1954f4c962bedca05ed5353fb0662b911d7fe548ac655f72073c461c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:875da3ddf07352f16bb090a9043d8a187f5b84c1f649bc20b2b48a8b00155bd5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da2db623f1232a72ff4178da60ca3829fae53651a89d703c8ab6de9d5d3d1b73"]}]
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest4
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2092024382493822976
n/a2091661989980934144
n/a2091299562508521472
n/a2090937171316838400
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e311.54%3 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 3 · Aug 21: 6 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:140]: failed to create HCP cluster "cp-ystream-upgrade-4-21-xbp8kt" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000d57060>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-xbp8kt: failed waiting for cluster="cp-ystream-upgrade-4-21-xbp8kt" in resourcegroup="rg-cp-ystream-upgrade-4-21-mvbldsj9x64s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0b50fbb-5e08-405c-bc27-320fd18ae286
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0b50fbb-5e08-405c-bc27-320fd18ae286",
      "name": "d0b50fbb-5e08-405c-bc27-320fd18ae286",
      "status": "Failed",
      "startTime": "2026-08-21T02:33:00.457364861Z",
      "endTime": "2026-08-21T02:52:03.680608781Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, control-plane-pki-operator, kube-scheduler, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, ignition-server, olm-operator, machine-approver, olm-collect-profiles, openshift-apiserver, cluster-policy-controller, csi-snapshot-controller-operator, catalog-operator, packageserver, community-operators-catalog, cluster-autoscaler, cluster-version-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, ignition-server-proxy, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:140]: failed to create HCP cluster "cp-ystream-upgrade-4-21-xbp8kt" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000d57060>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-xbp8kt: failed waiting for cluster="cp-ystream-upgrade-4-21-xbp8kt" in resourcegroup="rg-cp-ystream-upgrade-4-21-mvbldsj9x64s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0b50fbb-5e08-405c-bc27-320fd18ae286
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0b50fbb-5e08-405c-bc27-320fd18ae286",
      "name": "d0b50fbb-5e08-405c-bc27-320fd18ae286",
      "status": "Failed",
      "startTime": "2026-08-21T02:33:00.457364861Z",
      "endTime": "2026-08-21T02:52:03.680608781Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, control-plane-pki-operator, kube-scheduler, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, ignition-server, olm-operator, machine-approver, olm-collect-profiles, openshift-apiserver, cluster-policy-controller, csi-snapshot-controller-operator, catalog-operator, packageserver, community-operators-catalog, cluster-autoscaler, cluster-version-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, ignition-server-proxy, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f0f080>: 
    failed to create HCP cluster np-downgrade-kjw6pt: failed waiting for cluster="np-downgrade-kjw6pt" in resourcegroup="rg-np-downgrade-kjw6pt-lf69f9frnjjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98f96261-7c0c-4604-82d2-ccf83d040db2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98f96261-7c0c-4604-82d2-ccf83d040db2",
      "name": "98f96261-7c0c-4604-82d2-ccf83d040db2",
      "status": "Failed",
      "startTime": "2026-08-21T02:48:21.62730862Z",
      "endTime": "2026-08-21T03:07:24.73168706Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-route-controller-manager, cluster-policy-controller, ingress-operator, cluster-image-registry-operator, certified-operators-catalog, konnectivity-agent, catalog-operator, redhat-operators-catalog, control-plane-pki-operator, hosted-cluster-config-operator, packageserver, community-operators-catalog, kube-scheduler, azure-cloud-controller-manager, openshift-apiserver, cluster-version-operator, dns-operator, olm-operator, olm-collect-profiles, kube-controller-manager, cluster-network-operator, redhat-marketplace-catalog, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:581]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f0f080>: 
    failed to create HCP cluster np-downgrade-kjw6pt: failed waiting for cluster="np-downgrade-kjw6pt" in resourcegroup="rg-np-downgrade-kjw6pt-lf69f9frnjjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98f96261-7c0c-4604-82d2-ccf83d040db2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98f96261-7c0c-4604-82d2-ccf83d040db2",
      "name": "98f96261-7c0c-4604-82d2-ccf83d040db2",
      "status": "Failed",
      "startTime": "2026-08-21T02:48:21.62730862Z",
      "endTime": "2026-08-21T03:07:24.73168706Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-route-controller-manager, cluster-policy-controller, ingress-operator, cluster-image-registry-operator, certified-operators-catalog, konnectivity-agent, catalog-operator, redhat-operators-catalog, control-plane-pki-operator, hosted-cluster-config-operator, packageserver, community-operators-catalog, kube-scheduler, azure-cloud-controller-manager, openshift-apiserver, cluster-version-operator, dns-operator, olm-operator, olm-collect-profiles, kube-controller-manager, cluster-network-operator, redhat-marketplace-catalog, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000fbc300>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dnkv5vg5t2ts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/abfca9a0-da31-4fda-b740-f92e856a7dca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/abfca9a0-da31-4fda-b740-f92e856a7dca",
      "name": "abfca9a0-da31-4fda-b740-f92e856a7dca",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:36.161779463Z",
      "endTime": "2026-08-21T02:22:42.321981114Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-node-tuning-operator, csi-snapshot-controller-operator, ignition-server-proxy, cluster-policy-controller, azure-cloud-controller-manager, cluster-storage-operator, konnectivity-agent, olm-collect-profiles, redhat-marketplace-catalog, certified-operators-catalog, packageserver, community-operators-catalog, machine-approver, control-plane-pki-operator, ignition-server, openshift-controller-manager, dns-operator, cluster-autoscaler, kube-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-network-operator, cluster-version-operator, ingress-operator, cluster-image-registry-operator, catalog-operator, olm-operator, openshift-apiserver, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000fbc300>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dnkv5vg5t2ts" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/abfca9a0-da31-4fda-b740-f92e856a7dca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/abfca9a0-da31-4fda-b740-f92e856a7dca",
      "name": "abfca9a0-da31-4fda-b740-f92e856a7dca",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:36.161779463Z",
      "endTime": "2026-08-21T02:22:42.321981114Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-node-tuning-operator, csi-snapshot-controller-operator, ignition-server-proxy, cluster-policy-controller, azure-cloud-controller-manager, cluster-storage-operator, konnectivity-agent, olm-collect-profiles, redhat-marketplace-catalog, certified-operators-catalog, packageserver, community-operators-catalog, machine-approver, control-plane-pki-operator, ignition-server, openshift-controller-manager, dns-operator, cluster-autoscaler, kube-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-network-operator, cluster-version-operator, ingress-operator, cluster-image-registry-operator, catalog-operator, olm-operator, openshift-apiserver, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2090619934307520512
n/a2090419881110409216
n/a2090356957637513216
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.R...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e311.54%3 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 2 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de13e0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-2kbvdf4gpb7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158",
      "name": "1ea7be1d-979f-43d0-a6eb-4fadf5b52158",
      "status": "Failed",
      "startTime": "2026-08-20T09:49:43.253613436Z",
      "endTime": "2026-08-20T10:08:52.449510653Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de13e0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-2kbvdf4gpb7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ea7be1d-979f-43d0-a6eb-4fadf5b52158",
      "name": "1ea7be1d-979f-43d0-a6eb-4fadf5b52158",
      "status": "Failed",
      "startTime": "2026-08-20T09:49:43.253613436Z",
      "endTime": "2026-08-20T10:08:52.449510653Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000678260>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-lvmrprf9szxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659",
      "name": "dc452150-a44c-4cdf-95f9-3ab1762bc659",
      "status": "Failed",
      "startTime": "2026-08-20T03:43:58.925008531Z",
      "endTime": "2026-08-20T04:03:00.154367927Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000678260>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-lvmrprf9szxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc452150-a44c-4cdf-95f9-3ab1762bc659",
      "name": "dc452150-a44c-4cdf-95f9-3ab1762bc659",
      "status": "Failed",
      "startTime": "2026-08-20T03:43:58.925008531Z",
      "endTime": "2026-08-20T04:03:00.154367927Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ff4e60>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-l88qjllvzfng" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c412073-0e2e-4d8b-bffc-e50b877db2c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c412073-0e2e-4d8b-bffc-e50b877db2c8",
      "name": "3c412073-0e2e-4d8b-bffc-e50b877db2c8",
      "status": "Failed",
      "startTime": "2026-08-19T14:26:29.558969897Z",
      "endTime": "2026-08-19T14:45:35.6247326Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ff4e60>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-l88qjllvzfng" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c412073-0e2e-4d8b-bffc-e50b877db2c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c412073-0e2e-4d8b-bffc-e50b877db2c8",
      "name": "3c412073-0e2e-4d8b-bffc-e50b877db2c8",
      "status": "Failed",
      "startTime": "2026-08-19T14:26:29.558969897Z",
      "endTime": "2026-08-19T14:45:35.6247326Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2090356957637513216
n/a2090276754840096768
n/a2090064577516539904
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e27.69%2 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 2 · Aug 23: 0 · Aug 24: 0DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000886800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-7qtvldntpktk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ad306216-5622-4121-8bf1-f015261fa044
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ad306216-5622-4121-8bf1-f015261fa044",
      "name": "ad306216-5622-4121-8bf1-f015261fa044",
      "status": "Failed",
      "startTime": "2026-08-22T02:03:58.055302295Z",
      "endTime": "2026-08-22T02:23:05.756072788Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000886800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-7qtvldntpktk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ad306216-5622-4121-8bf1-f015261fa044
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ad306216-5622-4121-8bf1-f015261fa044",
      "name": "ad306216-5622-4121-8bf1-f015261fa044",
      "status": "Failed",
      "startTime": "2026-08-22T02:03:58.055302295Z",
      "endTime": "2026-08-22T02:23:05.756072788Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000517020>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9zbjjzhbwsc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8fbb6b40-a57a-44dc-9ea6-468eaf28a2fe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8fbb6b40-a57a-44dc-9ea6-468eaf28a2fe",
      "name": "8fbb6b40-a57a-44dc-9ea6-468eaf28a2fe",
      "status": "Failed",
      "startTime": "2026-08-22T02:03:58.196286523Z",
      "endTime": "2026-08-22T02:23:05.606457627Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000517020>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9zbjjzhbwsc7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8fbb6b40-a57a-44dc-9ea6-468eaf28a2fe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8fbb6b40-a57a-44dc-9ea6-468eaf28a2fe",
      "name": "8fbb6b40-a57a-44dc-9ea6-468eaf28a2fe",
      "status": "Failed",
      "startTime": "2026-08-22T02:03:58.196286523Z",
      "endTime": "2026-08-22T02:23:05.606457627Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0008e61c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gdl8mt5sbkqm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4841240d-e2f8-4813-8ee4-75401f65cda9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4841240d-e2f8-4813-8ee4-75401f65cda9",
      "name": "4841240d-e2f8-4813-8ee4-75401f65cda9",
      "status": "Failed",
      "startTime": "2026-08-21T02:05:01.793828852Z",
      "endTime": "2026-08-21T02:24:02.290641543Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0008e61c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-gdl8mt5sbkqm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4841240d-e2f8-4813-8ee4-75401f65cda9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4841240d-e2f8-4813-8ee4-75401f65cda9",
      "name": "4841240d-e2f8-4813-8ee4-75401f65cda9",
      "status": "Failed",
      "startTime": "2026-08-21T02:05:01.793828852Z",
      "endTime": "2026-08-21T02:24:02.290641543Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090982470185914368
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e27.69%2 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 2 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-4r45c4hp7cql/cluster-candidate-4-20-r96znq should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008fa700>: 
    failed to create HCP cluster cluster-candidate-4-20-r96znq: failed waiting for cluster="cluster-candidate-4-20-r96znq" in resourcegroup="rg-candidate-4-20-4r45c4hp7cql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e",
      "name": "3742f6f7-818c-4bbe-a0c4-ca70030eb73e",
      "status": "Failed",
      "startTime": "2026-08-20T13:15:18.696546382Z",
      "endTime": "2026-08-20T13:34:20.875186386Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-4r45c4hp7cql/cluster-candidate-4-20-r96znq should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008fa700>: 
    failed to create HCP cluster cluster-candidate-4-20-r96znq: failed waiting for cluster="cluster-candidate-4-20-r96znq" in resourcegroup="rg-candidate-4-20-4r45c4hp7cql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3742f6f7-818c-4bbe-a0c4-ca70030eb73e",
      "name": "3742f6f7-818c-4bbe-a0c4-ca70030eb73e",
      "status": "Failed",
      "startTime": "2026-08-20T13:15:18.696546382Z",
      "endTime": "2026-08-20T13:34:20.875186386Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-gpmtwfgcr6xn/cluster-candidate-4-20-6mrg8j should provision
Unexpected error:
    <*fmt.wrapError | 0xc001090160>: 
    failed to create HCP cluster cluster-candidate-4-20-6mrg8j: failed waiting for cluster="cluster-candidate-4-20-6mrg8j" in resourcegroup="rg-candidate-4-20-gpmtwfgcr6xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4",
      "name": "a1f59ecc-759d-4b6c-8b93-932b0208adf4",
      "status": "Failed",
      "startTime": "2026-08-20T08:53:33.518746975Z",
      "endTime": "2026-08-20T09:12:39.900608486Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-gpmtwfgcr6xn/cluster-candidate-4-20-6mrg8j should provision
Unexpected error:
    <*fmt.wrapError | 0xc001090160>: 
    failed to create HCP cluster cluster-candidate-4-20-6mrg8j: failed waiting for cluster="cluster-candidate-4-20-6mrg8j" in resourcegroup="rg-candidate-4-20-gpmtwfgcr6xn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1f59ecc-759d-4b6c-8b93-932b0208adf4",
      "name": "a1f59ecc-759d-4b6c-8b93-932b0208adf4",
      "status": "Failed",
      "startTime": "2026-08-20T08:53:33.518746975Z",
      "endTime": "2026-08-20T09:12:39.900608486Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090419881110409216
n/a2090356957637513216
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: externalAuths, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e27.69%2 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 0 · Aug 18: 1 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001a4a120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-drrz4q8s78wp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ae99f150-7046-4373-b424-e51c79b56a22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ae99f150-7046-4373-b424-e51c79b56a22",
      "name": "ae99f150-7046-4373-b424-e51c79b56a22",
      "status": "Failed",
      "startTime": "2026-08-19T14:30:35.01030213Z",
      "endTime": "2026-08-19T14:54:35.627231464Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9mlei3q3ci7sc5kdghjha0ako2p2s2 still exists (deletion dispatched at 2026-08-19T14:30:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001a4a120>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-drrz4q8s78wp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ae99f150-7046-4373-b424-e51c79b56a22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ae99f150-7046-4373-b424-e51c79b56a22",
      "name": "ae99f150-7046-4373-b424-e51c79b56a22",
      "status": "Failed",
      "startTime": "2026-08-19T14:30:35.01030213Z",
      "endTime": "2026-08-19T14:54:35.627231464Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9mlei3q3ci7sc5kdghjha0ako2p2s2 still exists (deletion dispatched at 2026-08-19T14:30:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011702b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-dcntnkvt776v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948f6139-f188-494a-b1a0-5f0f1e06acaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948f6139-f188-494a-b1a0-5f0f1e06acaf",
      "name": "948f6139-f188-494a-b1a0-5f0f1e06acaf",
      "status": "Failed",
      "startTime": "2026-08-19T00:47:39.330861834Z",
      "endTime": "2026-08-19T01:11:43.18871778Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ai28ppvdqu27hrgup8c3fclsl6ao7 still exists (deletion dispatched at 2026-08-19T00:47:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011702b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-dcntnkvt776v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948f6139-f188-494a-b1a0-5f0f1e06acaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/948f6139-f188-494a-b1a0-5f0f1e06acaf",
      "name": "948f6139-f188-494a-b1a0-5f0f1e06acaf",
      "status": "Failed",
      "startTime": "2026-08-19T00:47:39.330861834Z",
      "endTime": "2026-08-19T01:11:43.18871778Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s9ai28ppvdqu27hrgup8c3fclsl6ao7 still exists (deletion dispatched at 2026-08-19T00:47:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/externalAuths, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2090064577516539904
n/a2089849852383989760
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube con...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsAug 17: 12 · Aug 18: 47 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000b26d20>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-rxs7l9wkbj9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aceae8b1-ad97-4fd9-85eb-b3b27c9aebb7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aceae8b1-ad97-4fd9-85eb-b3b27c9aebb7",
      "name": "aceae8b1-ad97-4fd9-85eb-b3b27c9aebb7",
      "status": "Failed",
      "startTime": "2026-08-18T02:02:45.64795053Z",
      "endTime": "2026-08-18T02:21:50.493813123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000b26d20>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-rxs7l9wkbj9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aceae8b1-ad97-4fd9-85eb-b3b27c9aebb7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aceae8b1-ad97-4fd9-85eb-b3b27c9aebb7",
      "name": "aceae8b1-ad97-4fd9-85eb-b3b27c9aebb7",
      "status": "Failed",
      "startTime": "2026-08-18T02:02:45.64795053Z",
      "endTime": "2026-08-18T02:21:50.493813123Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fa8840>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-zdptt7lrhq2v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ed61913-8f7b-4249-8b7b-d53652b4a639
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ed61913-8f7b-4249-8b7b-d53652b4a639",
      "name": "8ed61913-8f7b-4249-8b7b-d53652b4a639",
      "status": "Failed",
      "startTime": "2026-08-18T03:33:58.580313157Z",
      "endTime": "2026-08-18T03:53:04.009688419Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fa8840>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-zdptt7lrhq2v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ed61913-8f7b-4249-8b7b-d53652b4a639
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ed61913-8f7b-4249-8b7b-d53652b4a639",
      "name": "8ed61913-8f7b-4249-8b7b-d53652b4a639",
      "status": "Failed",
      "startTime": "2026-08-18T03:33:58.580313157Z",
      "endTime": "2026-08-18T03:53:04.009688419Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000b151a0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-5vv87vj8ff2q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/09fc3a7a-6044-4ad3-857e-0da0e928149d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/09fc3a7a-6044-4ad3-857e-0da0e928149d",
      "name": "09fc3a7a-6044-4ad3-857e-0da0e928149d",
      "status": "Failed",
      "startTime": "2026-08-18T02:49:17.430704094Z",
      "endTime": "2026-08-18T03:08:22.123216944Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000b151a0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-5vv87vj8ff2q" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/09fc3a7a-6044-4ad3-857e-0da0e928149d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/09fc3a7a-6044-4ad3-857e-0da0e928149d",
      "name": "09fc3a7a-6044-4ad3-857e-0da0e928149d",
      "status": "Failed",
      "startTime": "2026-08-18T02:49:17.430704094Z",
      "endTime": "2026-08-18T03:08:22.123216944Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (47)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260630preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.191
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089532925895774208
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 8 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc001a80bd0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a6ef0ec-688d-48ac-96fb-22ede87538f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a6ef0ec-688d-48ac-96fb-22ede87538f5",
      "name": "0a6ef0ec-688d-48ac-96fb-22ede87538f5",
      "status": "Failed",
      "startTime": "2026-08-21T02:04:28.992322331Z",
      "endTime": "2026-08-21T02:23:32.540875768Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc001a80bd0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a6ef0ec-688d-48ac-96fb-22ede87538f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a6ef0ec-688d-48ac-96fb-22ede87538f5",
      "name": "0a6ef0ec-688d-48ac-96fb-22ede87538f5",
      "status": "Failed",
      "startTime": "2026-08-21T02:04:28.992322331Z",
      "endTime": "2026-08-21T02:23:32.540875768Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cc1940>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-p8kxn92zjqhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5185baff-cb0e-4088-baa6-c6571cadd51f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5185baff-cb0e-4088-baa6-c6571cadd51f",
      "name": "5185baff-cb0e-4088-baa6-c6571cadd51f",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:36.75245159Z",
      "endTime": "2026-08-21T02:22:42.448207861Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cc1940>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-p8kxn92zjqhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5185baff-cb0e-4088-baa6-c6571cadd51f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5185baff-cb0e-4088-baa6-c6571cadd51f",
      "name": "5185baff-cb0e-4088-baa6-c6571cadd51f",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:36.75245159Z",
      "endTime": "2026-08-21T02:22:42.448207861Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000edcea0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k8fl8dx2bw22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42d6143a-e97b-4483-afd9-f74bce1b7086
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42d6143a-e97b-4483-afd9-f74bce1b7086",
      "name": "42d6143a-e97b-4483-afd9-f74bce1b7086",
      "status": "Failed",
      "startTime": "2026-08-21T02:44:46.647452675Z",
      "endTime": "2026-08-21T03:03:54.737093461Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000edcea0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k8fl8dx2bw22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42d6143a-e97b-4483-afd9-f74bce1b7086
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/42d6143a-e97b-4483-afd9-f74bce1b7086",
      "name": "42d6143a-e97b-4483-afd9-f74bce1b7086",
      "status": "Failed",
      "startTime": "2026-08-21T02:44:46.647452675Z",
      "endTime": "2026-08-21T03:03:54.737093461Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (8)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider M...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 8 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000cc9300>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-6cppfxw8g4qh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5d7c5e-2952-42c3-bb62-e707bfb32ce1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5d7c5e-2952-42c3-bb62-e707bfb32ce1",
      "name": "1e5d7c5e-2952-42c3-bb62-e707bfb32ce1",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:28.697047474Z",
      "endTime": "2026-08-21T02:22:32.526910358Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000cc9300>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-6cppfxw8g4qh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5d7c5e-2952-42c3-bb62-e707bfb32ce1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e5d7c5e-2952-42c3-bb62-e707bfb32ce1",
      "name": "1e5d7c5e-2952-42c3-bb62-e707bfb32ce1",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:28.697047474Z",
      "endTime": "2026-08-21T02:22:32.526910358Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f84520>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fq28q2k4mqnr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7022aa18-ac7c-4bd8-b67a-9fb51bb8cb51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7022aa18-ac7c-4bd8-b67a-9fb51bb8cb51",
      "name": "7022aa18-ac7c-4bd8-b67a-9fb51bb8cb51",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:19.705086647Z",
      "endTime": "2026-08-21T02:22:22.572455462Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f84520>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fq28q2k4mqnr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7022aa18-ac7c-4bd8-b67a-9fb51bb8cb51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7022aa18-ac7c-4bd8-b67a-9fb51bb8cb51",
      "name": "7022aa18-ac7c-4bd8-b67a-9fb51bb8cb51",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:19.705086647Z",
      "endTime": "2026-08-21T02:22:22.572455462Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-wlltck9mnpdh/cluster-candidate-4-21-wqbh9c should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e0c460>: 
    failed to create HCP cluster cluster-candidate-4-21-wqbh9c: failed waiting for cluster="cluster-candidate-4-21-wqbh9c" in resourcegroup="rg-candidate-4-21-wlltck9mnpdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61fe2608-61d1-400b-90c7-806bb5489f2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61fe2608-61d1-400b-90c7-806bb5489f2b",
      "name": "61fe2608-61d1-400b-90c7-806bb5489f2b",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:28.583859524Z",
      "endTime": "2026-08-21T02:22:32.536144456Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-wlltck9mnpdh/cluster-candidate-4-21-wqbh9c should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e0c460>: 
    failed to create HCP cluster cluster-candidate-4-21-wqbh9c: failed waiting for cluster="cluster-candidate-4-21-wqbh9c" in resourcegroup="rg-candidate-4-21-wlltck9mnpdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61fe2608-61d1-400b-90c7-806bb5489f2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/61fe2608-61d1-400b-90c7-806bb5489f2b",
      "name": "61fe2608-61d1-400b-90c7-806bb5489f2b",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:28.583859524Z",
      "endTime": "2026-08-21T02:22:32.536144456Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (8)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-5jbtkc
Unexpected error:
    <*fmt.wrapError | 0xc00147aa60>: 
    failed to create HCP cluster np-noedge-5jbtkc: failed waiting for cluster="np-noedge-5jbtkc" in resourcegroup="rg-np-noedge-5jbtkc-5kwghxcpjkf6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/edc3030e-e575-42e1-8662-b09b51002c15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/edc3030e-e575-42e1-8662-b09b51002c15",
      "name": "edc3030e-e575-42e1-8662-b09b51002c15",
      "status": "Failed",
      "startTime": "2026-08-21T02:45:54.462061981Z",
      "endTime": "2026-08-21T03:04:54.752602351Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, ingress-operator, cluster-image-registry-operator, catalog-operator, dns-operator, packageserver, redhat-operators-catalog, community-operators-catalog, cluster-version-operator, cluster-policy-controller, certified-operators-catalog, openshift-route-controller-manager, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-network-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create HCP cluster np-noedge-5jbtkc
Unexpected error:
    <*fmt.wrapError | 0xc00147aa60>: 
    failed to create HCP cluster np-noedge-5jbtkc: failed waiting for cluster="np-noedge-5jbtkc" in resourcegroup="rg-np-noedge-5jbtkc-5kwghxcpjkf6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/edc3030e-e575-42e1-8662-b09b51002c15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/edc3030e-e575-42e1-8662-b09b51002c15",
      "name": "edc3030e-e575-42e1-8662-b09b51002c15",
      "status": "Failed",
      "startTime": "2026-08-21T02:45:54.462061981Z",
      "endTime": "2026-08-21T03:04:54.752602351Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, ingress-operator, cluster-image-registry-operator, catalog-operator, dns-operator, packageserver, redhat-operators-catalog, community-operators-catalog, cluster-version-operator, cluster-policy-controller, certified-operators-catalog, openshift-route-controller-manager, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-network-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005fe2a0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-ttg7gz2lhlcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba59ea70-9b35-4783-a0ec-868e44666cde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba59ea70-9b35-4783-a0ec-868e44666cde",
      "name": "ba59ea70-9b35-4783-a0ec-868e44666cde",
      "status": "Failed",
      "startTime": "2026-08-21T02:50:23.471170015Z",
      "endTime": "2026-08-21T03:09:24.769917396Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, dns-operator, catalog-operator, cluster-image-registry-operator, community-operators-catalog, openshift-route-controller-manager, olm-operator, cluster-policy-controller, certified-operators-catalog, cluster-version-operator, openshift-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:217]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005fe2a0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-ttg7gz2lhlcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba59ea70-9b35-4783-a0ec-868e44666cde
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ba59ea70-9b35-4783-a0ec-868e44666cde",
      "name": "ba59ea70-9b35-4783-a0ec-868e44666cde",
      "status": "Failed",
      "startTime": "2026-08-21T02:50:23.471170015Z",
      "endTime": "2026-08-21T03:09:24.769917396Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, dns-operator, catalog-operator, cluster-image-registry-operator, community-operators-catalog, openshift-route-controller-manager, olm-operator, cluster-policy-controller, certified-operators-catalog, cluster-version-operator, openshift-controller-manager, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-m6dffq9dg6px/cluster-candidate-4-22-rttbvz should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fee820>: 
    failed to create HCP cluster cluster-candidate-4-22-rttbvz: failed waiting for cluster="cluster-candidate-4-22-rttbvz" in resourcegroup="rg-candidate-4-22-m6dffq9dg6px" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6621367e-101e-4d5a-ab22-9e61d95865dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6621367e-101e-4d5a-ab22-9e61d95865dd",
      "name": "6621367e-101e-4d5a-ab22-9e61d95865dd",
      "status": "Failed",
      "startTime": "2026-08-21T02:29:22.397457374Z",
      "endTime": "2026-08-21T02:48:23.657141881Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-m6dffq9dg6px/cluster-candidate-4-22-rttbvz should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fee820>: 
    failed to create HCP cluster cluster-candidate-4-22-rttbvz: failed waiting for cluster="cluster-candidate-4-22-rttbvz" in resourcegroup="rg-candidate-4-22-m6dffq9dg6px" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6621367e-101e-4d5a-ab22-9e61d95865dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6621367e-101e-4d5a-ab22-9e61d95865dd",
      "name": "6621367e-101e-4d5a-ab22-9e61d95865dd",
      "status": "Failed",
      "startTime": "2026-08-21T02:29:22.397457374Z",
      "endTime": "2026-08-21T02:48:23.657141881Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-dqp9mm5cksqn/cluster-candidate-4-20-bfprqt should provision
Unexpected error:
    <*fmt.wrapError | 0xc0015cf400>: 
    failed to create HCP cluster cluster-candidate-4-20-bfprqt: failed waiting for cluster="cluster-candidate-4-20-bfprqt" in resourcegroup="rg-candidate-4-20-dqp9mm5cksqn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e423bc99-f4f6-4437-a051-cbb783aa9c19
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e423bc99-f4f6-4437-a051-cbb783aa9c19",
      "name": "e423bc99-f4f6-4437-a051-cbb783aa9c19",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:18.115131874Z",
      "endTime": "2026-08-21T02:22:22.541705225Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-dqp9mm5cksqn/cluster-candidate-4-20-bfprqt should provision
Unexpected error:
    <*fmt.wrapError | 0xc0015cf400>: 
    failed to create HCP cluster cluster-candidate-4-20-bfprqt: failed waiting for cluster="cluster-candidate-4-20-bfprqt" in resourcegroup="rg-candidate-4-20-dqp9mm5cksqn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e423bc99-f4f6-4437-a051-cbb783aa9c19
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e423bc99-f4f6-4437-a051-cbb783aa9c19",
      "name": "e423bc99-f4f6-4437-a051-cbb783aa9c19",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:18.115131874Z",
      "endTime": "2026-08-21T02:22:22.541705225Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 2 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, INT, PROD
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-h762z2 with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc000916f80>: 
    failed to create HCP cluster np-version-upgrade-cluster-h762z2: failed waiting for cluster="np-version-upgrade-cluster-h762z2" in resourcegroup="rg-np-version-upgrade-h762z2-227jm54s5fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a8fe4db-8a24-46bf-99be-8771ce79a814
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a8fe4db-8a24-46bf-99be-8771ce79a814",
      "name": "7a8fe4db-8a24-46bf-99be-8771ce79a814",
      "status": "Failed",
      "startTime": "2026-08-21T02:45:47.941159859Z",
      "endTime": "2026-08-21T03:04:54.892784877Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-h762z2 with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc000916f80>: 
    failed to create HCP cluster np-version-upgrade-cluster-h762z2: failed waiting for cluster="np-version-upgrade-cluster-h762z2" in resourcegroup="rg-np-version-upgrade-h762z2-227jm54s5fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a8fe4db-8a24-46bf-99be-8771ce79a814
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a8fe4db-8a24-46bf-99be-8771ce79a814",
      "name": "7a8fe4db-8a24-46bf-99be-8771ce79a814",
      "status": "Failed",
      "startTime": "2026-08-21T02:45:47.941159859Z",
      "endTime": "2026-08-21T03:04:54.892784877Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000c2e7a0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dhdl29xp87ff" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94b79bca-856d-47d2-b8ff-0c4f5c24fa7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94b79bca-856d-47d2-b8ff-0c4f5c24fa7b",
      "name": "94b79bca-856d-47d2-b8ff-0c4f5c24fa7b",
      "status": "Failed",
      "startTime": "2026-08-21T02:39:56.502029488Z",
      "endTime": "2026-08-21T02:59:03.450785892Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000c2e7a0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dhdl29xp87ff" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94b79bca-856d-47d2-b8ff-0c4f5c24fa7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94b79bca-856d-47d2-b8ff-0c4f5c24fa7b",
      "name": "94b79bca-856d-47d2-b8ff-0c4f5c24fa7b",
      "status": "Failed",
      "startTime": "2026-08-21T02:39:56.502029488Z",
      "endTime": "2026-08-21T02:59:03.450785892Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 2 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:228]: failed to update node pool np-1 with new label and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002a1d1a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-l7xnjldf8s7s" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-l7xnjldf8s7s, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:228]: failed to update node pool np-1 with new label and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002a1d1a0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-l7xnjldf8s7s" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-l7xnjldf8s7s, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:196]: failed to upgrade node pool npupgrade-4-20 to version 4.21.0-0.nightly-multi-2026-08-20-065923
Unexpected error:
    <*fmt.wrapErrors | 0xc002d08a80>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-z5466s" resourcegroup="rg-np-version-upgrade-z5466s-wkz757zpgbsr" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-z5466s in resource group rg-np-version-upgrade-z5466s-wkz757zpgbsr, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:196]: failed to upgrade node pool npupgrade-4-20 to version 4.21.0-0.nightly-multi-2026-08-20-065923
Unexpected error:
    <*fmt.wrapErrors | 0xc002d08a80>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-z5466s" resourcegroup="rg-np-version-upgrade-z5466s-wkz757zpgbsr" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-z5466s in resource group rg-np-version-upgrade-z5466s-wkz757zpgbsr, error: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090574634805104640
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url>
full failure pattern:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url>
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 2 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc000b06150>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout
    VerifyBasicAccess failed: failed to list services: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-20-052229, targetMinor=4.21.0-0.nightly-multi-2026-08-20-065923) failed: failed to get clusterversion "version": Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout
    VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.joinError | 0xc000b06150>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout
    VerifyBasicAccess failed: failed to list services: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-08-20-052229, targetMinor=4.21.0-0.nightly-multi-2026-08-20-065923) failed: failed to get clusterversion "version": Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout
    VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.l4p1f9f5i1x6x0a.jubs.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f0c138>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22.0-0.nightly-multi-2026-08-20-141954, targetMinor=4.23.0-0.nightly-multi-2026-08-20-085020) failed: failed to get clusterversion "version": Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout
    VerifyBasicAccess failed: failed to list services: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout
    VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:183]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f0c138>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": net/http: TLS handshake timeout
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.22.0-0.nightly-multi-2026-08-20-141954, targetMinor=4.23.0-0.nightly-multi-2026-08-20-085020) failed: failed to get clusterversion "version": Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": net/http: TLS handshake timeout
    VerifyBasicAccess failed: failed to list services: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": net/http: TLS handshake timeout
    VerifyKubeAPIServerServerVersionUpgraded failed: get kube-apiserver ServerVersion: Get "https://api.p4k3x7x1w2y0n2w.c3ss.uksouthstaging.aroapp-hcp.io:443/version": net/http: TLS handshake timeout
    ...
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090574634805104640
list nodes (nodePool=<nodepool>): Get "<url>
full failure pattern:
list nodes (nodePool=<nodepool>): Get "<url>
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 2 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:772]: Timed out after 2700.000s.
node pool nodes were not recreated at downgrade target version 4.20.0-0.nightly-multi-2026-08-20-052229
Expected success, but got an error:
    <*fmt.wrapError | 0xc000df20a0>: 
    list nodes (nodePool=npdg-422-420): Get "https://api.d1i3c9v1k0g4b5v.7gi9.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:772]: Timed out after 2700.000s.
node pool nodes were not recreated at downgrade target version 4.20.0-0.nightly-multi-2026-08-20-052229
Expected success, but got an error:
    <*fmt.wrapError | 0xc000df20a0>: 
    list nodes (nodePool=npdg-422-420): Get "https://api.d1i3c9v1k0g4b5v.7gi9.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0010e2920>: 
    list nodes (nodePool=nps-420-422): Get "https://api.m1w5a4x7b2n1l4i.t4j1.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": EOF
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:493]: Timed out after 2700.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0010e2920>: 
    list nodes (nodePool=nps-420-422): Get "https://api.m1w5a4x7b2n1l4i.t4j1.uksouthstaging.aroapp-hcp.io:443/api/v1/nodes": EOF
    ...
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090574634805104640
Cluster provisioning failed
full failure pattern:
Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsAug 17: 2 · Aug 18: 1 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2089532925895774208
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsAug 17: 2 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-stz97h" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0005b4620>: 
    failed to create HCP cluster cluster-zstream-4-21-stz97h: failed waiting for cluster="cluster-zstream-4-21-stz97h" in resourcegroup="rg-zstream-upgrade-4-21-jr9rvssp6jdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc3c97f5-82ee-48a8-bb99-d6a0a63dc350
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc3c97f5-82ee-48a8-bb99-d6a0a63dc350",
      "name": "dc3c97f5-82ee-48a8-bb99-d6a0a63dc350",
      "status": "Failed",
      "startTime": "2026-08-21T02:32:46.784949272Z",
      "endTime": "2026-08-21T02:51:53.64362497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, ingress-operator, cluster-version-operator, cluster-policy-controller, dns-operator, cluster-image-registry-operator, csi-snapshot-controller-operator, catalog-operator, packageserver, olm-operator, redhat-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, kube-controller-manager, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-stz97h" with version 4.21.28 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0005b4620>: 
    failed to create HCP cluster cluster-zstream-4-21-stz97h: failed waiting for cluster="cluster-zstream-4-21-stz97h" in resourcegroup="rg-zstream-upgrade-4-21-jr9rvssp6jdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc3c97f5-82ee-48a8-bb99-d6a0a63dc350
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc3c97f5-82ee-48a8-bb99-d6a0a63dc350",
      "name": "dc3c97f5-82ee-48a8-bb99-d6a0a63dc350",
      "status": "Failed",
      "startTime": "2026-08-21T02:32:46.784949272Z",
      "endTime": "2026-08-21T02:51:53.64362497Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, ingress-operator, cluster-version-operator, cluster-policy-controller, dns-operator, cluster-image-registry-operator, csi-snapshot-controller-operator, catalog-operator, packageserver, olm-operator, redhat-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, kube-controller-manager, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:146]: failed to delete customer resource group "cx-rg-cluster-z2b7swxgrmhl"
Unexpected error:
    <*fmt.wrapErrors | 0xc0039aa420>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-z2b7swxgrmhl, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:146]: failed to delete customer resource group "cx-rg-cluster-z2b7swxgrmhl"
Unexpected error:
    <*fmt.wrapErrors | 0xc0039aa420>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-z2b7swxgrmhl, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degra...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0011a6420>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wpsgmw5rph44" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb23f88c-ec78-4c60-8fbf-d5847d914873
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb23f88c-ec78-4c60-8fbf-d5847d914873",
      "name": "eb23f88c-ec78-4c60-8fbf-d5847d914873",
      "status": "Failed",
      "startTime": "2026-08-21T02:05:11.172903386Z",
      "endTime": "2026-08-21T02:24:12.542645954Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0011a6420>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wpsgmw5rph44" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb23f88c-ec78-4c60-8fbf-d5847d914873
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb23f88c-ec78-4c60-8fbf-d5847d914873",
      "name": "eb23f88c-ec78-4c60-8fbf-d5847d914873",
      "status": "Failed",
      "startTime": "2026-08-21T02:05:11.172903386Z",
      "endTime": "2026-08-21T02:24:12.542645954Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:140]: failed to create HCP cluster "cp-ystream-upgrade-4-22-52rvxg" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0018c2cc0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-52rvxg: failed waiting for cluster="cp-ystream-upgrade-4-22-52rvxg" in resourcegroup="rg-cp-ystream-upgrade-4-22-sht9lcl99cmr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ab73772b-6b4b-4639-a6b8-b8a05b2d2e3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ab73772b-6b4b-4639-a6b8-b8a05b2d2e3a",
      "name": "ab73772b-6b4b-4639-a6b8-b8a05b2d2e3a",
      "status": "Failed",
      "startTime": "2026-08-21T02:33:22.674679438Z",
      "endTime": "2026-08-21T02:52:23.424354393Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, cluster-storage-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, kube-controller-manager, kube-scheduler, cluster-policy-controller, csi-snapshot-controller-operator, cluster-version-operator, openshift-route-controller-manager, catalog-operator, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-network-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:140]: failed to create HCP cluster "cp-ystream-upgrade-4-22-52rvxg" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0018c2cc0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-52rvxg: failed waiting for cluster="cp-ystream-upgrade-4-22-52rvxg" in resourcegroup="rg-cp-ystream-upgrade-4-22-sht9lcl99cmr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ab73772b-6b4b-4639-a6b8-b8a05b2d2e3a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ab73772b-6b4b-4639-a6b8-b8a05b2d2e3a",
      "name": "ab73772b-6b4b-4639-a6b8-b8a05b2d2e3a",
      "status": "Failed",
      "startTime": "2026-08-21T02:33:22.674679438Z",
      "endTime": "2026-08-21T02:52:23.424354393Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, cluster-storage-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, community-operators-catalog, kube-controller-manager, kube-scheduler, cluster-policy-controller, csi-snapshot-controller-operator, cluster-version-operator, openshift-route-controller-manager, catalog-operator, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-network-operator, olm-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-sttt5r at version 4.22.10
Unexpected error:
    <*fmt.wrapError | 0xc000f130e0>: 
    failed to create HCP cluster np-skip-minor-sttt5r: failed waiting for cluster="np-skip-minor-sttt5r" in resourcegroup="rg-np-skip-minor-sttt5r-4fsvkfwk5vdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa",
      "name": "a570b418-a849-489b-8b7e-8e8fb8135baa",
      "status": "Failed",
      "startTime": "2026-08-20T09:36:05.972495231Z",
      "endTime": "2026-08-20T09:55:11.507838131Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, azure-cloud-controller-manager, dns-operator, olm-operator, machine-approver, kube-controller-manager, hosted-cluster-config-operator, kube-storage-version-migrator, catalog-operator, olm-collect-profiles, kube-scheduler, community-operators-catalog, cluster-autoscaler, openshift-route-controller-manager, cluster-policy-controller, csi-snapshot-controller-operator, packageserver, control-plane-pki-operator, cluster-storage-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, openshift-apiserver, cluster-image-registry-operator, redhat-marketplace-catalog, certified-operators-catalog, openshift-controller-manager, cluster-network-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:442]: failed to create HCP cluster np-skip-minor-sttt5r at version 4.22.10
Unexpected error:
    <*fmt.wrapError | 0xc000f130e0>: 
    failed to create HCP cluster np-skip-minor-sttt5r: failed waiting for cluster="np-skip-minor-sttt5r" in resourcegroup="rg-np-skip-minor-sttt5r-4fsvkfwk5vdh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a570b418-a849-489b-8b7e-8e8fb8135baa",
      "name": "a570b418-a849-489b-8b7e-8e8fb8135baa",
      "status": "Failed",
      "startTime": "2026-08-20T09:36:05.972495231Z",
      "endTime": "2026-08-20T09:55:11.507838131Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, azure-cloud-controller-manager, dns-operator, olm-operator, machine-approver, kube-controller-manager, hosted-cluster-config-operator, kube-storage-version-migrator, catalog-operator, olm-collect-profiles, kube-scheduler, community-operators-catalog, cluster-autoscaler, openshift-route-controller-manager, cluster-policy-controller, csi-snapshot-controller-operator, packageserver, control-plane-pki-operator, cluster-storage-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, openshift-apiserver, cluster-image-registry-operator, redhat-marketplace-catalog, certified-operators-catalog, openshift-controller-manager, cluster-network-operator, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090356957637513216
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver, router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0012312e0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mm9t5htdp974" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e464b7e0-c213-4e6e-988f-74f58142aa41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e464b7e0-c213-4e6e-988f-74f58142aa41",
      "name": "e464b7e0-c213-4e6e-988f-74f58142aa41",
      "status": "Failed",
      "startTime": "2026-08-21T02:05:47.205015937Z",
      "endTime": "2026-08-21T02:24:52.289709904Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, packageserver, redhat-marketplace-catalog, openshift-controller-manager, ingress-operator, olm-operator, redhat-operators-catalog, certified-operators-catalog, cluster-storage-operator, csi-snapshot-controller-operator, cluster-node-tuning-operator, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-policy-controller, cluster-network-operator, catalog-operator, community-operators-catalog, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0012312e0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mm9t5htdp974" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e464b7e0-c213-4e6e-988f-74f58142aa41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e464b7e0-c213-4e6e-988f-74f58142aa41",
      "name": "e464b7e0-c213-4e6e-988f-74f58142aa41",
      "status": "Failed",
      "startTime": "2026-08-21T02:05:47.205015937Z",
      "endTime": "2026-08-21T02:24:52.289709904Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, packageserver, redhat-marketplace-catalog, openshift-controller-manager, ingress-operator, olm-operator, redhat-operators-catalog, certified-operators-catalog, cluster-storage-operator, csi-snapshot-controller-operator, cluster-node-tuning-operator, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-policy-controller, cluster-network-operator, catalog-operator, community-operators-catalog, dns-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, openshift-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000dba740>: 
    failed to create HCP cluster ea-listfncws6: failed waiting for cluster="ea-listfncws6" in resourcegroup="ea-list-zhgflbxgws4b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04c8d3ea-1f82-496d-aa9a-800028c0f1a6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04c8d3ea-1f82-496d-aa9a-800028c0f1a6",
      "name": "04c8d3ea-1f82-496d-aa9a-800028c0f1a6",
      "status": "Failed",
      "startTime": "2026-08-21T02:35:24.937663979Z",
      "endTime": "2026-08-21T02:54:33.431541939Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000dba740>: 
    failed to create HCP cluster ea-listfncws6: failed waiting for cluster="ea-listfncws6" in resourcegroup="ea-list-zhgflbxgws4b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04c8d3ea-1f82-496d-aa9a-800028c0f1a6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/04c8d3ea-1f82-496d-aa9a-800028c0f1a6",
      "name": "04c8d3ea-1f82-496d-aa9a-800028c0f1a6",
      "status": "Failed",
      "startTime": "2026-08-21T02:35:24.937663979Z",
      "endTime": "2026-08-21T02:54:33.431541939Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, kube-scheduler, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0028eeae0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-92dczdzgqxgg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/739a7635-b32c-4f38-8193-2975abc5e092
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/739a7635-b32c-4f38-8193-2975abc5e092",
      "name": "739a7635-b32c-4f38-8193-2975abc5e092",
      "status": "Failed",
      "startTime": "2026-08-21T02:40:33.871209807Z",
      "endTime": "2026-08-21T02:59:33.678476291Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0028eeae0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-92dczdzgqxgg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/739a7635-b32c-4f38-8193-2975abc5e092
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/739a7635-b32c-4f38-8193-2975abc5e092",
      "name": "739a7635-b32c-4f38-8193-2975abc5e092",
      "status": "Failed",
      "startTime": "2026-08-21T02:40:33.871209807Z",
      "endTime": "2026-08-21T02:59:33.678476291Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001344800>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-q78wj4tcrbbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31dc1e1d-a696-4310-84a7-03cd3b7f433b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31dc1e1d-a696-4310-84a7-03cd3b7f433b",
      "name": "31dc1e1d-a696-4310-84a7-03cd3b7f433b",
      "status": "Failed",
      "startTime": "2026-08-21T02:44:35.391627616Z",
      "endTime": "2026-08-21T03:03:43.616320763Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001344800>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-q78wj4tcrbbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31dc1e1d-a696-4310-84a7-03cd3b7f433b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/31dc1e1d-a696-4310-84a7-03cd3b7f433b",
      "name": "31dc1e1d-a696-4310-84a7-03cd3b7f433b",
      "status": "Failed",
      "startTime": "2026-08-21T02:44:35.391627616Z",
      "endTime": "2026-08-21T03:03:43.616320763Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc001178800>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-njrg4blgvtnt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5481864c-7cad-43c3-99c6-7f83d731f12b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5481864c-7cad-43c3-99c6-7f83d731f12b",
      "name": "5481864c-7cad-43c3-99c6-7f83d731f12b",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:29.121350695Z",
      "endTime": "2026-08-21T02:22:32.49197336Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc001178800>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-njrg4blgvtnt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5481864c-7cad-43c3-99c6-7f83d731f12b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5481864c-7cad-43c3-99c6-7f83d731f12b",
      "name": "5481864c-7cad-43c3-99c6-7f83d731f12b",
      "status": "Failed",
      "startTime": "2026-08-21T02:03:29.121350695Z",
      "endTime": "2026-08-21T02:22:32.49197336Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010f8e20>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-9qvbx4pls9rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/668bdee1-fe77-422b-b467-4884f4f94057
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/668bdee1-fe77-422b-b467-4884f4f94057",
      "name": "668bdee1-fe77-422b-b467-4884f4f94057",
      "status": "Failed",
      "startTime": "2026-08-19T13:34:19.958633732Z",
      "endTime": "2026-08-19T13:53:23.640928655Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:184]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0010f8e20>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-9qvbx4pls9rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/668bdee1-fe77-422b-b467-4884f4f94057
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/668bdee1-fe77-422b-b467-4884f4f94057",
      "name": "668bdee1-fe77-422b-b467-4884f4f94057",
      "status": "Failed",
      "startTime": "2026-08-19T13:34:19.958633732Z",
      "endTime": "2026-08-19T13:53:23.640928655Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090064577516539904
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000e9e8e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wlkvs7868jnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e159d376-391e-45cb-a50d-4792a4a230e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e159d376-391e-45cb-a50d-4792a4a230e8",
      "name": "e159d376-391e-45cb-a50d-4792a4a230e8",
      "status": "Failed",
      "startTime": "2026-08-19T13:32:05.123641839Z",
      "endTime": "2026-08-19T13:51:13.689263263Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:106]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000e9e8e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-wlkvs7868jnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e159d376-391e-45cb-a50d-4792a4a230e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e159d376-391e-45cb-a50d-4792a4a230e8",
      "name": "e159d376-391e-45cb-a50d-4792a4a230e8",
      "status": "Failed",
      "startTime": "2026-08-19T13:32:05.123641839Z",
      "endTime": "2026-08-19T13:51:13.689263263Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelFIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090064577516539904
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e68800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-rddwcsq6xrbg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce",
      "name": "bf0b0684-dd10-4cf3-87c3-baf48258b7ce",
      "status": "Failed",
      "startTime": "2026-08-20T13:16:38.793143062Z",
      "endTime": "2026-08-20T13:35:40.482871637Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e68800>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-rddwcsq6xrbg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf0b0684-dd10-4cf3-87c3-baf48258b7ce",
      "name": "bf0b0684-dd10-4cf3-87c3-baf48258b7ce",
      "status": "Failed",
      "startTime": "2026-08-20T13:16:38.793143062Z",
      "endTime": "2026-08-20T13:35:40.482871637Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090419881110409216
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, machine-approver, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000806660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2c57lt79tp2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e60c22e9-c32c-4219-b8d7-30131e20f6f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e60c22e9-c32c-4219-b8d7-30131e20f6f1",
      "name": "e60c22e9-c32c-4219-b8d7-30131e20f6f1",
      "status": "Failed",
      "startTime": "2026-08-19T13:33:35.958939424Z",
      "endTime": "2026-08-19T13:52:43.668779021Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.m3b8f2l5y3g1d7d.320w.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000806660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-2c57lt79tp2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e60c22e9-c32c-4219-b8d7-30131e20f6f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e60c22e9-c32c-4219-b8d7-30131e20f6f1",
      "name": "e60c22e9-c32c-4219-b8d7-30131e20f6f1",
      "status": "Failed",
      "startTime": "2026-08-19T13:33:35.958939424Z",
      "endTime": "2026-08-19T13:52:43.668779021Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.m3b8f2l5y3g1d7d.320w.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090064577516539904
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, ignition-server, redhat-marketplace-catalog]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7af60>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-kfjt8dvwt56j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12",
      "name": "54f121a6-620a-4f93-bac1-3e84405aac12",
      "status": "Failed",
      "startTime": "2026-08-20T09:44:43.862276988Z",
      "endTime": "2026-08-20T10:03:52.490075448Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.a8j1x1l8p8y5l8a.lecf.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:270]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f7af60>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-kfjt8dvwt56j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/54f121a6-620a-4f93-bac1-3e84405aac12",
      "name": "54f121a6-620a-4f93-bac1-3e84405aac12",
      "status": "Failed",
      "startTime": "2026-08-20T09:44:43.862276988Z",
      "endTime": "2026-08-20T10:03:52.490075448Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.a8j1x1l8p8y5l8a.lecf.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090356957637513216
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable:...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, konnectivity-agent, kube-scheduler]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-5lt48h with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc001068a60>: 
    failed to create HCP cluster np-version-upgrade-cluster-5lt48h: failed waiting for cluster="np-version-upgrade-cluster-5lt48h" in resourcegroup="rg-np-version-upgrade-5lt48h-dszsrsbrq6zk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe765ae8-0a48-4400-a1c0-66c693e2b5fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe765ae8-0a48-4400-a1c0-66c693e2b5fa",
      "name": "fe765ae8-0a48-4400-a1c0-66c693e2b5fa",
      "status": "Failed",
      "startTime": "2026-08-21T02:45:36.662949608Z",
      "endTime": "2026-08-21T03:04:44.745401761Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.q5e3m9a0x8h7s0u.7eq2.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:147]: failed to create HCP cluster np-version-upgrade-cluster-5lt48h with version 4.21.29
Unexpected error:
    <*fmt.wrapError | 0xc001068a60>: 
    failed to create HCP cluster np-version-upgrade-cluster-5lt48h: failed waiting for cluster="np-version-upgrade-cluster-5lt48h" in resourcegroup="rg-np-version-upgrade-5lt48h-dszsrsbrq6zk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe765ae8-0a48-4400-a1c0-66c693e2b5fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fe765ae8-0a48-4400-a1c0-66c693e2b5fa",
      "name": "fe765ae8-0a48-4400-a1c0-66c693e2b5fa",
      "status": "Failed",
      "startTime": "2026-08-21T02:45:36.662949608Z",
      "endTime": "2026-08-21T03:04:44.745401761Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.q5e3m9a0x8h7s0u.7eq2.uksouthstaging.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000ce5680>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-s6nq8d7z6854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68d2c327-4b78-47bd-bbdb-95c6f476f1ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68d2c327-4b78-47bd-bbdb-95c6f476f1ab",
      "name": "68d2c327-4b78-47bd-bbdb-95c6f476f1ab",
      "status": "Failed",
      "startTime": "2026-08-21T02:04:33.015430331Z",
      "endTime": "2026-08-21T02:23:42.504347676Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000ce5680>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-s6nq8d7z6854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68d2c327-4b78-47bd-bbdb-95c6f476f1ab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/68d2c327-4b78-47bd-bbdb-95c6f476f1ab",
      "name": "68d2c327-4b78-47bd-bbdb-95c6f476f1ab",
      "status": "Failed",
      "startTime": "2026-08-21T02:04:33.015430331Z",
      "endTime": "2026-08-21T02:23:42.504347676Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 1 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001048e40>: 
    failed to create HCP cluster np-dg-minor-dc29s7: failed waiting for cluster="np-dg-minor-dc29s7" in resourcegroup="rg-np-dg-minor-dc29s7-9gmklftjh28x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6bbe4242-1a21-482f-8b6d-fe7e97ed8cb2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6bbe4242-1a21-482f-8b6d-fe7e97ed8cb2",
      "name": "6bbe4242-1a21-482f-8b6d-fe7e97ed8cb2",
      "status": "Failed",
      "startTime": "2026-08-21T02:49:33.691417488Z",
      "endTime": "2026-08-21T03:08:34.727253217Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:721]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001048e40>: 
    failed to create HCP cluster np-dg-minor-dc29s7: failed waiting for cluster="np-dg-minor-dc29s7" in resourcegroup="rg-np-dg-minor-dc29s7-9gmklftjh28x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6bbe4242-1a21-482f-8b6d-fe7e97ed8cb2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/6bbe4242-1a21-482f-8b6d-fe7e97ed8cb2",
      "name": "6bbe4242-1a21-482f-8b6d-fe7e97ed8cb2",
      "status": "Failed",
      "startTime": "2026-08-21T02:49:33.691417488Z",
      "endTime": "2026-08-21T03:08:34.727253217Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090619934307520512
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] <no_message>; [clusterServiceStatus] <no_message>; ...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] <no_message>; [clusterServiceStatus] <no_message>; [descendantResources] <no_message>; [hostedCluster] <no_message>; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 1DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001716270>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-nml9dcvzzfr9" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e8f4e5-f6d1-44fe-818c-64543efd65cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e8f4e5-f6d1-44fe-818c-64543efd65cf",
      "name": "d0e8f4e5-f6d1-44fe-818c-64543efd65cf",
      "status": "Failed",
      "startTime": "2026-08-24T23:14:11.789973414Z",
      "endTime": "2026-08-24T23:38:15.678890881Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc001716270>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-nml9dcvzzfr9" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e8f4e5-f6d1-44fe-818c-64543efd65cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d0e8f4e5-f6d1-44fe-818c-64543efd65cf",
      "name": "d0e8f4e5-f6d1-44fe-818c-64543efd65cf",
      "status": "Failed",
      "startTime": "2026-08-24T23:14:11.789973414Z",
      "endTime": "2026-08-24T23:38:15.678890881Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] \u003cno_message\u003e; [clusterServiceStatus] \u003cno_message\u003e; [descendantResources] \u003cno_message\u003e; [hostedCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should not be able to reuse subnets and NSGs between clusters1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2092024382493822976
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachabl...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message [controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 1 · Aug 20: 0 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-w8t8gk" to 4.23.0-0.nightly-multi-2026-08-19-140455
Unexpected error:
    <*fmt.wrapError | 0xc00160f3a0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-w8t8gk" in resourcegroup="rg-cp-ystream-upgrade-4-23-ldgb8b99w4xp" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97cf82cb-bb1f-4aac-aaea-96415f8aa42a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97cf82cb-bb1f-4aac-aaea-96415f8aa42a",
      "name": "97cf82cb-bb1f-4aac-aaea-96415f8aa42a",
      "status": "Failed",
      "startTime": "2026-08-19T23:45:11.57498354Z",
      "endTime": "2026-08-19T23:45:21.100010011Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:174]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-w8t8gk" to 4.23.0-0.nightly-multi-2026-08-19-140455
Unexpected error:
    <*fmt.wrapError | 0xc00160f3a0>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-w8t8gk" in resourcegroup="rg-cp-ystream-upgrade-4-23-ldgb8b99w4xp" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97cf82cb-bb1f-4aac-aaea-96415f8aa42a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/97cf82cb-bb1f-4aac-aaea-96415f8aa42a",
      "name": "97cf82cb-bb1f-4aac-aaea-96415f8aa42a",
      "status": "Failed",
      "startTime": "2026-08-19T23:45:11.57498354Z",
      "endTime": "2026-08-19T23:45:21.100010011Z",
      "error": {
        "code": "InvalidRequestContent",
        "message": "[controlPlaneDesiredVersionResolution] no upgrade path found from 4.22.0-0.nightly-multi-2026-08-17-010411 to 4.23.0: no reachable versions in target minor and no gateway version in current minor"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090212361167179776
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.34; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.20": must be at least 4.20.34; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-4lxbwp" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0011b1e00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p9w9mgdh26kn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-4lxbwp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.33",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.34",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-20-4lxbwp" to minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc0011b1e00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p9w9mgdh26kn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-4lxbwp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": may not decrease from 4.20.33",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.20\": must be at least 4.20.34",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090257540762832896
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.29; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "4.21": must be at least 4.21.29; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.85%1 of 26 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsAug 17: 0 · Aug 18: 0 · Aug 19: 0 · Aug 20: 1 · Aug 21: 0 · Aug 22: 0 · Aug 23: 0 · Aug 24: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-qkbdvr" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0008bcae0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qdqdx9dwcprj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-qkbdvr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.28",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.29",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:148]: failed to pin cluster "cluster-zstream-4-21-qkbdvr" to minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0008bcae0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qdqdx9dwcprj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-qkbdvr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": may not decrease from 4.21.28",
            "target": "properties.version.id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"4.21\": must be at least 4.21.29",
            "target": "properties.version.id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2090257540762832896