CIHealth
<
Single Day: Sep 26▾
Last 7 Days Weekly: Sep 21 - Sep 27 Sprint: Sep 14 - Sep 27 Single Day: Sep 26
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 21 · Failures matched: 101 · Total runs: 76 · Runs affected: 74

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-version-operator]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.32%1 of 76 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b86200>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9464kjvgqp5t" to finish creating: GET https://rp.j0737536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34a5d5ce-cb5b-4d49-9d05-cf14dfa067ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34a5d5ce-cb5b-4d49-9d05-cf14dfa067ae",
      "name": "34a5d5ce-cb5b-4d49-9d05-cf14dfa067ae",
      "status": "Failed",
      "startTime": "2026-09-26T22:26:42.363515696Z",
      "endTime": "2026-09-26T22:55:52.65701083Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, redhat-operators-catalog, certified-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b86200>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9464kjvgqp5t" to finish creating: GET https://rp.j0737536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34a5d5ce-cb5b-4d49-9d05-cf14dfa067ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34a5d5ce-cb5b-4d49-9d05-cf14dfa067ae",
      "name": "34a5d5ce-cb5b-4d49-9d05-cf14dfa067ae",
      "status": "Failed",
      "startTime": "2026-09-26T22:26:42.363515696Z",
      "endTime": "2026-09-26T22:55:52.65701083Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, redhat-operators-catalog, certified-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: cluster-version-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71852103965959260737536
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRrun command "e2e-runcommand-1790458991773693237" failed with exit code 28 on VM "<vm>" (output: "000")
full failure pattern:
run command "e2e-runcommand-1790458991773693237" failed with exit code 28 on VM "<vm>" (output: "000")
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.32%1 of 76 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 608.861s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00107dc50>: 
    run command "e2e-runcommand-1790458991773693237" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 608.861s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00107dc50>: 
    run command "e2e-runcommand-1790458991773693237" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #70682103942172993654784
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
full failure pattern:
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
provision6078.95%60 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 11 · Sep 25: 43 · Sep 26: 60none
Full failure examples (3)
time=2026-09-26T14:49:06.836Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T14:49:08.412Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T14:49:09.638Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=6089940dde784d763a773e4c9004101f74c31b1e2eea1a61564da81e9a0036e5 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5653504%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6089940dde784d763a773e4c9004101f74c31b1e2eea1a61564da81e9a0036e5
time=2026-09-26T14:52:40.545Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5653504/providers/Microsoft.Resources/deployments/6089940dde784d763a773e4c9004101f74c31b1e2eea1a61564da81e9a0036e5/operationStatuses/08584111727364954260\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5653504/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5653504/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j5653504\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 81aead43-b997-4ca0-92b1-5f890a98ae35:9/26/2026 2:50:21 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T14:54:40.554Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T14:54:41.206Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T14:54:42.164Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9881912e12f161c343a185ab325eab0f9959440aad9a04d75fdbf5669aaa2bef portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5653504%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9881912e12f161c343a185ab325eab0f9959440aad9a04d75fdbf5669aaa2bef
time=2026-09-26T15:10:45.962Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5653504/providers/Microsoft.Resources/deployments/9881912e12f161c343a185ab325eab0f9959440aad9a04d75fdbf5669aaa2bef/operationStatuses/08584111724039322810\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5653504/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5653504/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j5653504\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 7053750e-7976-4092-b5c9-05af8bee4bc0:9/26/2026 3:10:24 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T15:12:45.966Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T15:12:46.574Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T15:12:47.251Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=5c70e5fdb4253270876a8cf87e4ad360a842c5c82a499a0efaf8267c12be42a2 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5653504%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5c70e5fdb4253270876a8cf87e4ad360a842c5c82a499a0efaf8267c12be42a2
time=2026-09-26T15:19:06.585Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
time=2026-09-26T14:49:01.802Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T14:49:03.910Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T14:49:05.531Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9ec5d968c4a2abbfa09f2d13ea926004d6d393845d4e1c03095043f158076d84 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6813568%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9ec5d968c4a2abbfa09f2d13ea926004d6d393845d4e1c03095043f158076d84
time=2026-09-26T14:52:06.712Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6813568/providers/Microsoft.Resources/deployments/9ec5d968c4a2abbfa09f2d13ea926004d6d393845d4e1c03095043f158076d84/operationStatuses/08584111727409288089\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6813568/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6813568/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j6813568\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 3938f6fe-2589-4619-be28-fc66822767aa:9/26/2026 2:50:13 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T14:54:06.713Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T14:54:07.745Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T14:54:09.171Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=8afa4a83c65198c5a6152d3ce561b308250ec90817107c38ff6b5a156799c6da portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6813568%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8afa4a83c65198c5a6152d3ce561b308250ec90817107c38ff6b5a156799c6da
time=2026-09-26T15:09:44.252Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6813568/providers/Microsoft.Resources/deployments/8afa4a83c65198c5a6152d3ce561b308250ec90817107c38ff6b5a156799c6da/operationStatuses/08584111724372557912\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6813568/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6813568/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j6813568\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 381e25a6-705f-4967-8b86-0e72fd08d4dc:9/26/2026 3:09:17 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T15:11:44.253Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T15:11:45.072Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T15:11:46.506Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f8da50925da962c46c0feb369b1ccdd3215b84037bcc35598bc1f3f1121f0d5d portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6813568%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff8da50925da962c46c0feb369b1ccdd3215b84037bcc35598bc1f3f1121f0d5d
time=2026-09-26T15:19:01.486Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
time=2026-09-26T14:43:15.259Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T14:43:16.912Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T14:43:17.849Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=553cb6c6b3fded1accbd2852e1223b9f885add89fe9c1ca9b9ab9fd387804af3 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1104128%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F553cb6c6b3fded1accbd2852e1223b9f885add89fe9c1ca9b9ab9fd387804af3
time=2026-09-26T14:47:48.727Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1104128/providers/Microsoft.Resources/deployments/553cb6c6b3fded1accbd2852e1223b9f885add89fe9c1ca9b9ab9fd387804af3/operationStatuses/08584111730881613482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1104128/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1104128/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1104128\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 5fd50c6a-1737-4597-a037-368930da6cc9:9/26/2026 2:44:27 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T14:49:48.731Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T14:49:49.490Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T14:49:50.425Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=99ebb22d91f4358f020049c1b68a81eba2ea522f9bacb97496c9a72de75bf919 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1104128%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F99ebb22d91f4358f020049c1b68a81eba2ea522f9bacb97496c9a72de75bf919
time=2026-09-26T15:05:54.057Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra step=infra error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1104128/providers/Microsoft.Resources/deployments/99ebb22d91f4358f020049c1b68a81eba2ea522f9bacb97496c9a72de75bf919/operationStatuses/08584111726955290959\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1104128/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1104128/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j1104128\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 169a3df3-f8a3-4c96-9074-128137b133d8:9/26/2026 3:05:31 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-26T15:07:54.062Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-26T15:07:54.711Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-26T15:07:55.697Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=9b984337a990465b3d1ba4b559dcc1260b91bb6b32219cd0080e1d7d34f722df portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1104128%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9b984337a990465b3d1ba4b559dcc1260b91bb6b32219cd0080e1d7d34f722df
time=2026-09-26T15:13:14.833Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra60
Affected runs (60)
Date (UTC)Associated PRProw job
PR #71642103852034485653504
PR #70682103851390936813568
PR #71572103851378941104128
PR #71882103851381092782080
PR #68212103851372238606336
PR #71702103851355876626432
PR #71742103851334296932352
PR #71872103851316445974528
PR #71052103848603104579584
PR #65772103848790996815872
PR #71722103848898379386880
PR #71782103790396751482880
PR #71852103789910530985984
PR #68462103786893190631424
PR #71572103785005913214976
PR #71782103775301015179264
PR #71852103774808339648512
PR #71572103764124197457920
PR #71782103760196256927744
PR #71852103759712255217664
PR #71782103745097093877760
PR #71852103744610089046016
PR #71742103733651282857984
PR #71642103730519727411200
PR #71782103729999365279744
PR #71852103729510443651072
PR #71782103714900751683584
PR #71852103714409951006720
PR #66612103705397104742400
PR #71572103701911558426624
PR #71782103699803396378624
PR #71852103699317209436160
PR #70682103685753023238144
PR #71782103684698914951168
PR #71852103684219090767872
PR #71702103670738761289728
PR #71422103670723087175680
PR #71722103670718657990656
PR #71642103670704342831104
PR #71682103670699393552384
PR #71742103670647581315072
PR #71692103670621719236608
PR #71782103667365953671168
PR #71852103667354889097216
PR #71832103656984816390144
PR #71872103655450791645184
PR #71882103655428259844096
PR #65772103654924834312192
PR #67752103650148490940416
PR #71702103641099342974976
PR #71642103641079956901888
PR #70682103641065637548032
PR #71682103641066266693632
PR #71722103641060163981312
PR #71422103641026831847424
PR #71692103641014462844928
PR #71762103641015683387392
PR #71742103640998813896704
PR #71832103639551695130624
PR #71812103639238770692096
alert [svc] FrontendPathMedianLatency fired
full failure pattern:
alert [svc] FrontendPathMedianLatency fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
alert56.58%5 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 20: 5 · Sep 21: 20 · Sep 22: 21 · Sep 23: 22 · Sep 24: 24 · Sep 25: 30 · Sep 26: 5none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:18:12Z
  Ended:       2026-09-27T00:27:15Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6084992-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6084992-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T17:44:26Z
  Ended:       2026-09-26T17:50:28Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6814080-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6814080-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T17:44:51Z
  Ended:       2026-09-26T17:50:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7663616-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7663616-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
PR #71572103881469456814080
PR #71642103881459847663616
PR #68212103881428969197568
PR #70682103881405791473664
alert [hcp] userJourneyEtcdReadLatencyP991h5m fired
full failure pattern:
alert [hcp] userJourneyEtcdReadLatencyP991h5m fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
alert56.58%5 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 6 · Sep 26: 5none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:30:46Z
  Ended:       2026-09-27T00:51:51Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j9219200-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t317no0qn2vco31uu1nj119agt86hub-agg-cluster", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j9219200-mgmt-1
Namespace: ocm-arohcpci01-2t317no0qn2vco31uu1nj119agt86hub-agg-cluster
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T20:33:18Z
  Ended:       2026-09-26T20:58:20Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j6283520-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t2tnffjjbth0k2idnuevp5detncdatr-cilium-cluster", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j6283520-mgmt-2
Namespace: ocm-arohcpci01-2t2tnffjjbth0k2idnuevp5detncdatr-cilium-cluster
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T17:32:15Z
  Ended:       2026-09-26T17:39:15Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j0431360-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t2qrbqcgq901aevacbo6vkcduiahunj-acr-pull-create", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0431360-mgmt-2
Namespace: ocm-arohcpci01-2t2qrbqcgq901aevacbo6vkcduiahunj-acr-pull-create
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdReadLatencyP991h5m does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #71942103990456999219200
PR #71782103929857506283520
PR #71872103881473420431360
PR #71882103881450792161280
PR #68212103881428969197568
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e45.26%4 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 2 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 4none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #71852103929841903472640
PR #71642103927280047755264
PR #71572103927255972450304
PR #71572103881469456814080
alert [hcp] userJourneyEtcdReadLatencyP996h30m fired
full failure pattern:
alert [hcp] userJourneyEtcdReadLatencyP996h30m fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
alert45.26%4 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 5 · Sep 26: 4none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:33:47Z
  Ended:       2026-09-27T00:52:51Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j9219200-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t317no0qn2vco31uu1nj119agt86hub-agg-cluster", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j9219200-mgmt-1
Namespace: ocm-arohcpci01-2t317no0qn2vco31uu1nj119agt86hub-agg-cluster
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T20:37:19Z
  Ended:       2026-09-26T21:23:32Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j6283520-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t2tnffjjbth0k2idnuevp5detncdatr-cilium-cluster", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j6283520-mgmt-2
Namespace: ocm-arohcpci01-2t2tnffjjbth0k2idnuevp5detncdatr-cilium-cluster
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T17:35:15Z
  Ended:       2026-09-26T18:04:16Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j0431360-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t2qrbqcgq901aevacbo6vkcduiahunj-acr-pull-create", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0431360-mgmt-2
Namespace: ocm-arohcpci01-2t2qrbqcgq901aevacbo6vkcduiahunj-acr-pull-create
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdReadLatencyP996h30m does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #71942103990456999219200
PR #71782103929857506283520
PR #71872103881473420431360
PR #68212103881428969197568
alert [hcp] userJourneyEtcdWriteLatencyP991h5m fired
full failure pattern:
alert [hcp] userJourneyEtcdWriteLatencyP991h5m fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
alert33.95%3 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 5 · Sep 26: 3none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T23:57:44Z
  Ended:       2026-09-27T00:21:42Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j6084992-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t30min1009r4686mdn5h02sl9p5nb55-r4w4c9x5f5e5h4o", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j6084992-mgmt-2
Namespace: ocm-arohcpci01-2t30min1009r4686mdn5h02sl9p5nb55-r4w4c9x5f5e5h4o
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T22:38:39Z
  Ended:       2026-09-26T23:07:40Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j0737536-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t2viddtv02l63m6ul7m30kp2hdmu31l-agg-cluster", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0737536-mgmt-1
Namespace: ocm-arohcpci01-2t2viddtv02l63m6ul7m30kp2hdmu31l-agg-cluster
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T20:32:56Z
  Ended:       2026-09-26T20:58:54Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2450304-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t2tneh2r23glrlaq41sk5licb45u3pf-l2y8b5i4r2k8w3t", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2450304-mgmt-2
Namespace: ocm-arohcpci01-2t2tneh2r23glrlaq41sk5licb45u3pf-l2y8b5i4r2k8w3t

Firing 2:
  State:       Resolved
  Started:     2026-09-26T20:34:58Z
  Ended:       2026-09-26T20:56:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j2450304-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t2tna269g6d79a0hm4q37mmv6lq3q16-private-kas", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j2450304-mgmt-2
Namespace: ocm-arohcpci01-2t2tna269g6d79a0hm4q37mmv6lq3q16-private-kas
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdWriteLatencyP991h5m does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
PR #71852103965959260737536
PR #71572103927255972450304
alert [hcp] userJourneyEtcdWriteLatencyP996h30m fired
full failure pattern:
alert [hcp] userJourneyEtcdWriteLatencyP996h30m fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
alert33.95%3 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 4 · Sep 26: 3none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:00:39Z
  Ended:       2026-09-27T00:46:41Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j6084992-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t30min1009r4686mdn5h02sl9p5nb55-r4w4c9x5f5e5h4o", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j6084992-mgmt-2
Namespace: ocm-arohcpci01-2t30min1009r4686mdn5h02sl9p5nb55-r4w4c9x5f5e5h4o
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T22:41:41Z
  Ended:       2026-09-26T23:32:41Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j0737536-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t2viddtv02l63m6ul7m30kp2hdmu31l-agg-cluster", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0737536-mgmt-1
Namespace: ocm-arohcpci01-2t2viddtv02l63m6ul7m30kp2hdmu31l-agg-cluster
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T20:35:56Z
  Ended:       2026-09-26T21:23:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j2450304-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t2tneh2r23glrlaq41sk5licb45u3pf-l2y8b5i4r2k8w3t", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2450304-mgmt-2
Namespace: ocm-arohcpci01-2t2tneh2r23glrlaq41sk5licb45u3pf-l2y8b5i4r2k8w3t

Firing 2:
  State:       Resolved
  Started:     2026-09-26T20:37:57Z
  Ended:       2026-09-26T21:21:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j2450304-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t2tna269g6d79a0hm4q37mmv6lq3q16-private-kas", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j2450304-mgmt-2
Namespace: ocm-arohcpci01-2t2tna269g6d79a0hm4q37mmv6lq3q16-private-kas
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdWriteLatencyP996h30m does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
PR #71852103965959260737536
PR #71572103927255972450304
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
alert22.63%2 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 20: 1 · Sep 21: 1 · Sep 22: 3 · Sep 23: 2 · Sep 24: 4 · Sep 25: 1 · Sep 26: 2none
Full failure examples (2)
alert fired 4 time(s) (1 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:23:15Z
  Ended:       2026-09-27T00:29:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t30n01ijoaem1ekng72tj4pcrtjldgt-u0g5w9w4b6k9u1r", pod="kube-apiserver-6996784dc-8vzfd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t30n01ijoaem1ekng72tj4pcrtjldgt-u0g5w9w4b6k9u1r/kube-apiserver-6996784dc-8vzfd has been in a non-ready state for longer than 5 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T20:54:34Z
  Ended:       2026-09-26T21:00:35Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6283520-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t2u1irnugb4blbq80ksmoo4fkldqnt8-oidc-wi-cluster", pod="kube-controller-manager-65c84bdd67-wnnzz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t2u1irnugb4blbq80ksmoo4fkldqnt8-oidc-wi-cluster/kube-controller-manager-65c84bdd67-wnnzz has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-26T21:02:36Z
  Ended:       2026-09-26T21:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6283520-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t2tnr9frt52ianj6sih8n7ki9jvmnj1-v9n4d3d5t5m9z0v", pod="kube-apiserver-797f569f98-n962n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t2tnr9frt52ianj6sih8n7ki9jvmnj1-v9n4d3d5t5m9z0v/kube-apiserver-797f569f98-n962n has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
PR #71782103929857506283520
ERROR CODE: DeadlineExceeded; provider Microsoft.Network
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Network
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e22.63%2 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 2 · Sep 26: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wx6szx" for cluster "cluster-candidate-4-20-wx6szx"
Unexpected error:
    <*fmt.wrapError | 0xc000b82600>: 
    failed to create NodePool np-wx6szx: failed waiting for nodepool="np-wx6szx" for cluster "cluster-candidate-4-20-wx6szx" in resourcegroup="rg-candidate-4-20-r2kvmlfvs4fd" to finish creating: GET https://rp.j0737536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8ae21a1-086f-493e-86bc-e120b31e75a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8ae21a1-086f-493e-86bc-e120b31e75a1",
      "name": "a8ae21a1-086f-493e-86bc-e120b31e75a1",
      "status": "Failed",
      "startTime": "2026-09-26T22:38:07.749745969Z",
      "endTime": "2026-09-26T22:57:10.827139148Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x5q9i7c1r3h9e2l-np-wx6szx-s4276-jbfxc: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-20-r2kvmlfvs4fd-candidate-wx6szx--managed/x5q9i7c1r3h9e2l-np-wx6szx-s4276-jbfxc-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-r2kvmlfvs4fd-candidate-wx6szx--managed/providers/Microsoft.Network/networkInterfaces/x5q9i7c1r3h9e2l-np-wx6szx-s4276-jbfxc-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '10' ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wx6szx" for cluster "cluster-candidate-4-20-wx6szx"
Unexpected error:
    <*fmt.wrapError | 0xc000b82600>: 
    failed to create NodePool np-wx6szx: failed waiting for nodepool="np-wx6szx" for cluster "cluster-candidate-4-20-wx6szx" in resourcegroup="rg-candidate-4-20-r2kvmlfvs4fd" to finish creating: GET https://rp.j0737536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8ae21a1-086f-493e-86bc-e120b31e75a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8ae21a1-086f-493e-86bc-e120b31e75a1",
      "name": "a8ae21a1-086f-493e-86bc-e120b31e75a1",
      "status": "Failed",
      "startTime": "2026-09-26T22:38:07.749745969Z",
      "endTime": "2026-09-26T22:57:10.827139148Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x5q9i7c1r3h9e2l-np-wx6szx-s4276-jbfxc: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-20-r2kvmlfvs4fd-candidate-wx6szx--managed/x5q9i7c1r3h9e2l-np-wx6szx-s4276-jbfxc-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-r2kvmlfvs4fd-candidate-wx6szx--managed/providers/Microsoft.Network/networkInterfaces/x5q9i7c1r3h9e2l-np-wx6szx-s4276-jbfxc-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '10' ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vcbnqz" for cluster "cluster-candidate-4-20-vcbnqz"
Unexpected error:
    <*fmt.wrapError | 0xc000fe65e0>: 
    failed to create NodePool np-vcbnqz: failed waiting for nodepool="np-vcbnqz" for cluster "cluster-candidate-4-20-vcbnqz" in resourcegroup="rg-candidate-4-20-frjpqt6l6dlc" to finish creating: GET https://rp.j7663616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8abd2b48-24de-4fb1-bba7-e70679eac440
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8abd2b48-24de-4fb1-bba7-e70679eac440",
      "name": "8abd2b48-24de-4fb1-bba7-e70679eac440",
      "status": "Failed",
      "startTime": "2026-09-26T17:19:57.699169962Z",
      "endTime": "2026-09-26T17:38:57.270667313Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i2k9x4t6q6r7f1s-np-vcbnqz-qnsq9-j9kx5: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-20-frjpqt6l6dlc-candidate-vcbnqz--managed/i2k9x4t6q6r7f1s-np-vcbnqz-qnsq9-j9kx5-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-frjpqt6l6dlc-candidate-vcbnqz--managed/providers/Microsoft.Network/networkInterfaces/i2k9x4t6q6r7f1s-np-vcbnqz-qnsq9-j9kx5-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vcbnqz" for cluster "cluster-candidate-4-20-vcbnqz"
Unexpected error:
    <*fmt.wrapError | 0xc000fe65e0>: 
    failed to create NodePool np-vcbnqz: failed waiting for nodepool="np-vcbnqz" for cluster "cluster-candidate-4-20-vcbnqz" in resourcegroup="rg-candidate-4-20-frjpqt6l6dlc" to finish creating: GET https://rp.j7663616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8abd2b48-24de-4fb1-bba7-e70679eac440
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8abd2b48-24de-4fb1-bba7-e70679eac440",
      "name": "8abd2b48-24de-4fb1-bba7-e70679eac440",
      "status": "Failed",
      "startTime": "2026-09-26T17:19:57.699169962Z",
      "endTime": "2026-09-26T17:38:57.270667313Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i2k9x4t6q6r7f1s-np-vcbnqz-qnsq9-j9kx5: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-20-frjpqt6l6dlc-candidate-vcbnqz--managed/i2k9x4t6q6r7f1s-np-vcbnqz-qnsq9-j9kx5-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-frjpqt6l6dlc-candidate-vcbnqz--managed/providers/Microsoft.Network/networkInterfaces/i2k9x4t6q6r7f1s-np-vcbnqz-qnsq9-j9kx5-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71852103965959260737536
PR #71642103881459847663616
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 6 · Sep 26: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-9cp272" for cluster "cluster-candidate-4-23-9cp272"
Unexpected error:
    <*fmt.wrapError | 0xc00019bf20>: 
    failed to create NodePool np-9cp272: failed waiting for nodepool="np-9cp272" for cluster "cluster-candidate-4-23-9cp272" in resourcegroup="rg-candidate-4-23-t5nrns5dgzjz" to finish creating: GET https://rp.j6084992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92540fc4-c95a-4a5d-a847-5375c72243a8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92540fc4-c95a-4a5d-a847-5375c72243a8",
      "name": "92540fc4-c95a-4a5d-a847-5375c72243a8",
      "status": "Failed",
      "startTime": "2026-09-27T00:01:58.819547057Z",
      "endTime": "2026-09-27T00:21:07.730201043Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t5nrns5dgzjz-candidate-9cp272--managed/providers/Microsoft.Compute/virtualMachines/l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t5nrns5dgzjz-candidate-9cp272--managed/providers/Microsoft.Compute/virtualMachines/l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-9cp272" for cluster "cluster-candidate-4-23-9cp272"
Unexpected error:
    <*fmt.wrapError | 0xc00019bf20>: 
    failed to create NodePool np-9cp272: failed waiting for nodepool="np-9cp272" for cluster "cluster-candidate-4-23-9cp272" in resourcegroup="rg-candidate-4-23-t5nrns5dgzjz" to finish creating: GET https://rp.j6084992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92540fc4-c95a-4a5d-a847-5375c72243a8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/92540fc4-c95a-4a5d-a847-5375c72243a8",
      "name": "92540fc4-c95a-4a5d-a847-5375c72243a8",
      "status": "Failed",
      "startTime": "2026-09-27T00:01:58.819547057Z",
      "endTime": "2026-09-27T00:21:07.730201043Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t5nrns5dgzjz-candidate-9cp272--managed/providers/Microsoft.Compute/virtualMachines/l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t5nrns5dgzjz-candidate-9cp272--managed/providers/Microsoft.Compute/virtualMachines/l4k3w0h4y1l5l6t-np-9cp272-gmmfp-89h4j to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d2f360>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-9sjsgl" in resourcegroup="rg-np-version-upgrade-9sjsgl-z2gscsmxlxzq" to finish creating: GET https://rp.j6084992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ef95de3-9fbf-4193-8079-b1158861e584
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ef95de3-9fbf-4193-8079-b1158861e584",
      "name": "2ef95de3-9fbf-4193-8079-b1158861e584",
      "status": "Failed",
      "startTime": "2026-09-26T23:57:59.240971764Z",
      "endTime": "2026-09-27T00:17:06.977144117Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9sjsgl-z2gscsmxlxzq-np-upgrade-9s-96d5bba1/providers/Microsoft.Compute/virtualMachines/w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9sjsgl-z2gscsmxlxzq-np-upgrade-9s-96d5bba1/providers/Microsoft.Compute/virtualMachines/w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d2f360>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-9sjsgl" in resourcegroup="rg-np-version-upgrade-9sjsgl-z2gscsmxlxzq" to finish creating: GET https://rp.j6084992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ef95de3-9fbf-4193-8079-b1158861e584
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ef95de3-9fbf-4193-8079-b1158861e584",
      "name": "2ef95de3-9fbf-4193-8079-b1158861e584",
      "status": "Failed",
      "startTime": "2026-09-26T23:57:59.240971764Z",
      "endTime": "2026-09-27T00:17:06.977144117Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9sjsgl-z2gscsmxlxzq-np-upgrade-9s-96d5bba1/providers/Microsoft.Compute/virtualMachines/w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9sjsgl-z2gscsmxlxzq-np-upgrade-9s-96d5bba1/providers/Microsoft.Compute/virtualMachines/w5t3u0s9p5j2g6u-npupgrade-4-21-hgqjs-8fnck to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
alert11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 20: 4 · Sep 21: 2 · Sep 22: 2 · Sep 23: 2 · Sep 24: 4 · Sep 25: 6 · Sep 26: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-26T17:41:16Z
  Ended:       2026-09-26T17:47:10Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9197568-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t2qtvdnlaalvsj2ub667na9fqboc359-ephemeral-disk", pod="ignition-server-7fd4ccdc4c-psn2m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t2qtvdnlaalvsj2ub667na9fqboc359-ephemeral-disk/ignition-server-7fd4ccdc4c-psn2m has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68212103881428969197568
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
alert11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 20: 4 · Sep 21: 4 · Sep 22: 3 · Sep 23: 2 · Sep 24: 1 · Sep 25: 3 · Sep 26: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:06:56Z
  Ended:       2026-09-27T00:37:55Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z7blf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T00:06:56Z
  Ended:       2026-09-27T00:36:52Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z7blf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
alert11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsSep 20: 3 · Sep 21: 2 · Sep 22: 3 · Sep 23: 0 · Sep 24: 0 · Sep 25: 2 · Sep 26: 1none
Full failure examples (1)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-26T21:24:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2450304-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2450304/providers/microsoft.monitor/accounts/hcps-j2450304", namespace="ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w", pod="arohcp-monitor-kube-state-metrics-7988dc648c-q9544", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-26T21:24:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2450304-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2450304/providers/microsoft.monitor/accounts/hcps-j2450304", namespace="ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w", pod="arohcp-monitor-kube-state-metrics-7988dc648c-q9544", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-26T21:27:26Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2450304-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2450304/providers/microsoft.monitor/accounts/hcps-j2450304", namespace="ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w", pod="arohcp-monitor-kube-state-metrics-7988dc648c-q9544", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-26T21:27:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2450304-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2450304/providers/microsoft.monitor/accounts/hcps-j2450304", namespace="ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w", pod="arohcp-monitor-kube-state-metrics-7988dc648c-q9544", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t2tmvnf0viulta1u93uf2mlthhfuvsu-z7r2s8c1m6p3a8w/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71572103927255972450304
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)
alert11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)Show trend detailsSep 20: 1 · Sep 21: 2 · Sep 22: 2 · Sep 23: 1 · Sep 24: 0 · Sep 25: 1 · Sep 26: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:09:50Z
  Ended:       2026-09-27T00:37:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bb068423-1854-49c8-a132-0d10a6392326"
  Description: Pod ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-27T00:09:50Z
  Ended:       2026-09-27T00:37:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bb068423-1854-49c8-a132-0d10a6392326"
  Description: Pod ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
alert [svc] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [svc] KubeDeploymentReplicasMismatch fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
alert11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 20: 2 · Sep 21: 3 · Sep 22: 2 · Sep 23: 2 · Sep 24: 1 · Sep 25: 1 · Sep 26: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:21:19Z
  Ended:       2026-09-27T00:27:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/services-j6084992", namespace="klusterlet-2t30mivbmcgkot8vhp8oa13ibv1vnaut", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z7blf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t30mivbmcgkot8vhp8oa13ibv1vnaut/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
private ingress should not be reachable from outside the VNet, but connection succeeded
full failure pattern:
private ingress should not be reachable from outside the VNet, but connection succeeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #70682103881405791473664
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 2 · Sep 26: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00102e380>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-2x6vlrpgq2bh" to finish creating: GET https://rp.j6084992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47c0d2f0-4e66-4d5c-8e28-e145397d1b94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47c0d2f0-4e66-4d5c-8e28-e145397d1b94",
      "name": "47c0d2f0-4e66-4d5c-8e28-e145397d1b94",
      "status": "Failed",
      "startTime": "2026-09-26T23:46:18.825261842Z",
      "endTime": "2026-09-27T00:15:27.197366694Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00102e380>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-2x6vlrpgq2bh" to finish creating: GET https://rp.j6084992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47c0d2f0-4e66-4d5c-8e28-e145397d1b94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47c0d2f0-4e66-4d5c-8e28-e145397d1b94",
      "name": "47c0d2f0-4e66-4d5c-8e28-e145397d1b94",
      "status": "Failed",
      "startTime": "2026-09-26T23:46:18.825261842Z",
      "endTime": "2026-09-27T00:15:27.197366694Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782103981635216084992
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0020ae090>: 
    resource group "fips-enabled-k6r2vqf8b9xq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-k6r2vqf8b9xq" to finish deleting: GET https://rp.j2450304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50efbb08-7b80-41ad-a7a9-a51f5231b37f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50efbb08-7b80-41ad-a7a9-a51f5231b37f",
      "name": "50efbb08-7b80-41ad-a7a9-a51f5231b37f",
      "status": "Failed",
      "startTime": "2026-09-26T20:41:31.45560964Z",
      "endTime": "2026-09-26T21:05:35.034630666Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2tmvnf0viulta1u93uf2mlthhfuvsu still exists (deletion dispatched at 2026-09-26T20:41:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0020ae090>: 
    resource group "fips-enabled-k6r2vqf8b9xq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-k6r2vqf8b9xq" to finish deleting: GET https://rp.j2450304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50efbb08-7b80-41ad-a7a9-a51f5231b37f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/50efbb08-7b80-41ad-a7a9-a51f5231b37f",
      "name": "50efbb08-7b80-41ad-a7a9-a51f5231b37f",
      "status": "Failed",
      "startTime": "2026-09-26T20:41:31.45560964Z",
      "endTime": "2026-09-26T21:05:35.034630666Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2tmvnf0viulta1u93uf2mlthhfuvsu still exists (deletion dispatched at 2026-09-26T20:41:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71572103927255972450304
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.32%1 of 76 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 20: 0 · Sep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc002608b40>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-qqprr26gtjgf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qqprr26gtjgf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc002608b40>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-qqprr26gtjgf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-qqprr26gtjgf, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71642103927280047755264