CIHealth
<
Last 7 Days▾
Last 7 Days Weekly: Sep 14 - Sep 20 Sprint: Sep 14 - Sep 27 Single Day: Sep 16
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 68 · Failures matched: 343 · Total runs: 751 · Runs affected: 102

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] ArobitForwarderJobUp fired
full failure pattern:
alert [svc] ArobitForwarderJobUp fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert40.53%4 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 4none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T22:05:46Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j1475712-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j1475712-svc has been unreachable for 15 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:55:54Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j7836672-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j7836672-svc has been unreachable for 15 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T10:51:08Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j6352640-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j6352640-svc has been unreachable for 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ArobitForwarderJobUp does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #68202100332359231475712
PR #68202100270038647836672
PR #68202100161854826352640
PR #68202100132066686406656
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: ResourceGroupNotFound; provider Microsoft.Monitor
full failure pattern:
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Monitor
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e20.27%2 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc00072e820>: 
    failed to get workspace hcps-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/hcps-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc00072e820>: 
    failed to get workspace hcps-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/hcps-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-j3128064 in resource group hcp-underlay-ci01-j3128064
Unexpected error:
    <*fmt.wrapError | 0xc0010489e0>: 
    failed to get workspace services-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/services-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ama_metrics.go:83]: failed to look up SVC Prometheus endpoint for workspace services-j3128064 in resource group hcp-underlay-ci01-j3128064
Unexpected error:
    <*fmt.wrapError | 0xc0010489e0>: 
    failed to get workspace services-j3128064: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3128064/providers/Microsoft.Monitor/accounts/services-j3128064
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-j3128064' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc000c94e20>: 
    failed to get workspace hcps-yt: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.Monitor/accounts/hcps-yt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:74]: failed to look up HCP Prometheus endpoint
Unexpected error:
    <*fmt.wrapError | 0xc000c94e20>: 
    failed to get workspace hcps-yt: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-yt/providers/Microsoft.Monitor/accounts/hcps-yt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceGroupNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceGroupNotFound",
        "message": "Resource group 'hcp-underlay-ci01-yt' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAMA Metrics service metrics from SVC and MGMT clusters should be present in Azure Monitor2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelKSM HCP Metrics kube_node_info metrics should be present in Azure Monitor for the happy-path cluster2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #66722100194205623128064
PR #66722100129038159843328
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e20.27%2 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 2none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:183]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:183]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #63642100324480323686400
PR #63642100229912697769984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentStackDeploymentFailed
full failure pattern:
ERROR CODE: DeploymentStackDeploymentFailed
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision20.27%2 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 2none
Full failure examples (2)
time=2026-09-16T18:22:02.494Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-09-16T18:22:51.675Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/1df7c411-d9ac-443d-ac5d-b95681f8419b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/1df7c411-d9ac-443d-ac5d-b95681f8419b\",\n  \"name\": \"1df7c411-d9ac-443d-ac5d-b95681f8419b\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'd478c792-be07-45be-a743-97b4e94abf15'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5982976/providers/Microsoft.Resources/deployments/nbyngfosfawbsx5vtxb2uo6lmfim2rbaopp3emm2zwlmm-260916186gla8\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[0].Expression, Attempted value: null, Error: 1:237: parse error: unknown escape sequence U+002E '.' Activity ID: 95e8cbf4-5734-47ec-859e-d6ed81e0056f.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-09-16T17:10:05.170Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-09-16T17:11:24.952Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/aeca7e8d-50c1-47d0-a644-b291a00162ea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/canadacentral/deploymentStackOperationStatus/aeca7e8d-50c1-47d0-a644-b291a00162ea\",\n  \"name\": \"aeca7e8d-50c1-47d0-a644-b291a00162ea\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'df67a523-14d6-42a2-9636-444dbb7d2763'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2389376/providers/Microsoft.Resources/deployments/nbyngfosfawbsx5vtxb2uo6lmfim2rbaopp3emm2zwlmm-260916172yp3z\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[0].Expression, Attempted value: null, Error: 1:237: parse error: unknown escape sequence U+002E '.' Activity ID: 91890f9b-76b2-4791-9c30-927a7253a007.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Monitoring/regional/monitoring2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #69652100283069305982976
PR #69652100265336732389376
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to get kusto cluster <kusto-cluster> from config: configuration: key kusto not found
full failure pattern:
failed to get kusto cluster <kusto-cluster> from config: configuration: key kusto not found
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e20.27%2 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012e60a0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.000s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012e60a0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000a540c0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 600.003s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000a540c0>: 
    failed to get kusto cluster name from config: configuration: key kusto not found
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #66722100194205623128064
PR #66722100129038159843328
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to roll out Helm release: context deadline exceeded
full failure pattern:
failed to roll out Helm release: context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision20.27%2 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 2none
Full failure examples (2)
time=2026-09-16T10:35:02.273Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T10:35:02.716Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6352640-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6352640-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:ad132446-7529-417f-aff8-31d826ffb6f2 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#ad132446-7529-417f-aff8-31d826ffb6f2 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T10:35:02.716Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:35:02.863Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T10:35:03.042Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T10:35:03.042Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:35:03.143Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:45:05.949Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:45:05.949Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:46:05.955Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T10:46:06.637Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6352640-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6352640-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:ad132446-7529-417f-aff8-31d826ffb6f2 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#ad132446-7529-417f-aff8-31d826ffb6f2 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T10:46:06.637Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:07.013Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T10:46:07.408Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:07.408Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:07.670Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T10:46:08.488Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:08.489Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:08.888Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:09.401Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T10:46:09.747Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T10:46:09.959Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:10.361Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:10.729Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:46:11.065Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T10:46:11.065Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:46:11.148Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T10:56:12.973Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:56:12.973Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T10:57:12.977Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T10:57:13.486Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6352640-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6352640-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:ad132446-7529-417f-aff8-31d826ffb6f2 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#ad132446-7529-417f-aff8-31d826ffb6f2 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T10:57:13.486Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:13.807Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T10:57:14.174Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:14.174Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:14.404Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T10:57:15.329Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:15.329Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:15.688Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:15.987Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T10:57:16.345Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T10:57:16.535Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:16.747Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:16.984Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T10:57:17.208Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T10:57:17.208Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T10:57:17.300Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T11:05:02.358Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-16T11:05:02.359Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"
time=2026-09-16T08:43:27.720Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T08:43:28.143Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6406656-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6406656-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:510fcd49-73a6-4643-b104-d1da046c7a87 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#510fcd49-73a6-4643-b104-d1da046c7a87 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T08:43:28.143Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:43:28.283Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T08:43:28.434Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T08:43:28.434Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:43:28.493Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:53:31.202Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T08:53:31.203Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T08:54:31.204Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T08:54:31.661Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6406656-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6406656-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:510fcd49-73a6-4643-b104-d1da046c7a87 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#510fcd49-73a6-4643-b104-d1da046c7a87 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T08:54:31.661Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:31.919Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T08:54:32.413Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:32.413Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:32.686Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T08:54:33.497Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:33.498Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:33.902Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:34.404Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T08:54:34.757Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T08:54:34.953Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:35.182Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:35.544Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T08:54:36.068Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T08:54:36.068Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T08:54:36.176Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-16T09:04:37.727Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T09:04:37.727Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-16T09:05:37.733Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-16T09:05:38.373Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j6406656-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:94cee54eb85d08891b179bd74be48632bf6e7968f046225abb6074b4655891d0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:100m memory:256Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j6406656-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:0af23e7645c07968b9bd20f5c4a2966b34ccdaf8a05812b5eb5a3a26c53f24b5 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:510fcd49-73a6-4643-b104-d1da046c7a87 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#510fcd49-73a6-4643-b104-d1da046c7a87 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-16T09:05:38.373Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:38.676Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-16T09:05:39.023Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:39.023Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:39.181Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T09:05:40.018Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:40.018Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:40.264Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:40.639Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-16T09:05:41.058Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-16T09:05:41.352Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:41.683Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:42.063Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-16T09:05:42.276Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-16T09:05:42.276Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-16T09:05:42.359Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-16T09:13:27.645Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-16T09:13:27.646Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/arobit2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #68202100161854826352640
PR #68202100132066686406656
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRpermission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews
full failure pattern:
permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e20.27%2 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.100s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001341f60>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.100s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc001341f60>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.093s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000ff3040>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/volumeattachments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiextensions.k8s.io/customresourcedefinitions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list policy/poddisruptionbudgets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list autoscaling/horizontalpodautoscalers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list coordination.k8s.io/leases: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list discovery.k8s.io/endpointslices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list certificates.k8s.io/certificatesigningrequests: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list scheduling.k8s.io/priorityclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list node.k8s.io/runtimeclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/mutatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/validatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:480]: Timed out after 34.093s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc000ff3040>: 
    permission checks failed: SSAR failed for list /services: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /endpoints: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /events: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /persistentvolumeclaims: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /persistentvolumes: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /serviceaccounts: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /resourcequotas: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /limitranges: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list /replicationcontrollers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /componentstatuses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for get /pods/log: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/deployments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/daemonsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/statefulsets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apps/replicasets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/jobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list batch/cronjobs: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/networkpolicies: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list networking.k8s.io/ingresses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterroles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/clusterrolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/roles: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list rbac.authorization.k8s.io/rolebindings: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/storageclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list storage.k8s.io/volumeattachments: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list apiextensions.k8s.io/customresourcedefinitions: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list policy/poddisruptionbudgets: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list autoscaling/horizontalpodautoscalers: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list coordination.k8s.io/leases: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list discovery.k8s.io/endpointslices: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list certificates.k8s.io/certificatesigningrequests: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list scheduling.k8s.io/priorityclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list node.k8s.io/runtimeclasses: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/mutatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    SSAR failed for list admissionregistration.k8s.io/validatingwebhookconfigurations: the server could not find the requested resource (post selfsubjectaccessreviews.authorization.k8s.io)
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #63642100285588732121088
PR #63642100188464501755904
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000861120>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-pkbzl9cwqzmf" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823",
      "name": "7bc0c172-a0d6-40dc-821f-14c51cede823",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:59.260634412Z",
      "endTime": "2026-09-16T15:37:00.334559736Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000861120>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-pkbzl9cwqzmf" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7bc0c172-a0d6-40dc-821f-14c51cede823",
      "name": "7bc0c172-a0d6-40dc-821f-14c51cede823",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:59.260634412Z",
      "endTime": "2026-09-16T15:37:00.334559736Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:206]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc002a80740>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-czltd5xxrlvj" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "name": "180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "status": "Failed",
      "startTime": "2026-09-16T15:10:24.910079282Z",
      "endTime": "2026-09-16T15:39:30.319726782Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:206]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc002a80740>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-czltd5xxrlvj" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "name": "180c48f3-76f8-4d48-bcf4-3b4e29dffab7",
      "status": "Failed",
      "startTime": "2026-09-16T15:10:24.910079282Z",
      "endTime": "2026-09-16T15:39:30.319726782Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e96840>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-222ntj7gxqx2" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "name": "774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.983252228Z",
      "endTime": "2026-09-16T15:36:40.328092646Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:93]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e96840>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-222ntj7gxqx2" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "name": "774c9323-3626-4141-aa8c-a5f0d7e57e9c",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.983252228Z",
      "endTime": "2026-09-16T15:36:40.328092646Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #63642100229912697769984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining ...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 6none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0028a4438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j5zkl24x6bwt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827",
      "name": "5ecb9f85-876c-467d-91c9-980ce11c9827",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:19.070350042Z",
      "endTime": "2026-09-16T10:13:22.487477443Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0028a4438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j5zkl24x6bwt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ecb9f85-876c-467d-91c9-980ce11c9827",
      "name": "5ecb9f85-876c-467d-91c9-980ce11c9827",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:19.070350042Z",
      "endTime": "2026-09-16T10:13:22.487477443Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00153e0f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-9dwsgxpprw8c" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "name": "9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:48.885828252Z",
      "endTime": "2026-09-16T10:08:52.4506315Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00153e0f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-9dwsgxpprw8c" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "name": "9e39e5c6-0d40-4bf6-9634-0b867d3cf17c",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:48.885828252Z",
      "endTime": "2026-09-16T10:08:52.4506315Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012f78>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-g9bvp2" in resourcegroup="rg-candidate-4-20-pmdbfsk6v6zd" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429",
      "name": "834c2939-b42e-48bd-b340-960d8826a429",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:50.424611242Z",
      "endTime": "2026-09-16T10:13:52.492147666Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000012f78>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-g9bvp2" in resourcegroup="rg-candidate-4-20-pmdbfsk6v6zd" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/834c2939-b42e-48bd-b340-960d8826a429",
      "name": "834c2939-b42e-48bd-b340-960d8826a429",
      "status": "Failed",
      "startTime": "2026-09-16T09:49:50.424611242Z",
      "endTime": "2026-09-16T10:13:52.492147666Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892100141632635539456
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; prov...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:103]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001f3c120>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sxmn5zhcsvjn" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "name": "a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:36.527287031Z",
      "endTime": "2026-09-16T15:37:40.20476418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:103]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001f3c120>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sxmn5zhcsvjn" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "name": "a019abb2-ae16-4218-9707-6f7dc73a3eb3",
      "status": "Failed",
      "startTime": "2026-09-16T15:08:36.527287031Z",
      "endTime": "2026-09-16T15:37:40.20476418Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--7tj5b6r9d28k/cluster-hs-pre-rwztb2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ee48a0>: 
    failed to create HCP cluster cluster-hs-pre-rwztb2: failed waiting for cluster="cluster-hs-pre-rwztb2" in resourcegroup="rg-hs-pre--7tj5b6r9d28k" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d",
      "name": "0015f221-0420-4231-b96b-76ce90032f9d",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.092292569Z",
      "endTime": "2026-09-16T15:36:40.334174064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--7tj5b6r9d28k/cluster-hs-pre-rwztb2 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ee48a0>: 
    failed to create HCP cluster cluster-hs-pre-rwztb2: failed waiting for cluster="cluster-hs-pre-rwztb2" in resourcegroup="rg-hs-pre--7tj5b6r9d28k" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0015f221-0420-4231-b96b-76ce90032f9d",
      "name": "0015f221-0420-4231-b96b-76ce90032f9d",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:38.092292569Z",
      "endTime": "2026-09-16T15:36:40.334174064Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d8e640>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mldclqf9sr5h" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44",
      "name": "704efcc4-4b42-4704-9ff8-f01712701a44",
      "status": "Failed",
      "startTime": "2026-09-16T15:09:00.206878787Z",
      "endTime": "2026-09-16T15:38:00.28474108Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d8e640>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-mldclqf9sr5h" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/704efcc4-4b42-4704-9ff8-f01712701a44",
      "name": "704efcc4-4b42-4704-9ff8-f01712701a44",
      "status": "Failed",
      "startTime": "2026-09-16T15:09:00.206878787Z",
      "endTime": "2026-09-16T15:38:00.28474108Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #63642100229912697769984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001512108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-mwjj2q" in resourcegroup="admin-credential-lifecycle-test-q4n7ckbgk7dt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4",
      "name": "5124f997-7212-409c-99ae-fbd89a51e2f4",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:00.208113506Z",
      "endTime": "2026-09-16T10:08:02.509863824Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12vvsmbd0thdp0f88kc41u6pvi7c9 still exists (deletion dispatched at 2026-09-16T09:44:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001512108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-mwjj2q" in resourcegroup="admin-credential-lifecycle-test-q4n7ckbgk7dt" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5124f997-7212-409c-99ae-fbd89a51e2f4",
      "name": "5124f997-7212-409c-99ae-fbd89a51e2f4",
      "status": "Failed",
      "startTime": "2026-09-16T09:44:00.208113506Z",
      "endTime": "2026-09-16T10:08:02.509863824Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12vvsmbd0thdp0f88kc41u6pvi7c9 still exists (deletion dispatched at 2026-09-16T09:44:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008820c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-hz87cjqncdvb" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3",
      "name": "8e715185-33d5-494b-976c-25b2ab261cc3",
      "status": "Failed",
      "startTime": "2026-09-16T09:35:58.536453378Z",
      "endTime": "2026-09-16T10:00:02.536562643Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1371bjtto4nf5m6o0pid4pjvle6qc still exists (deletion dispatched at 2026-09-16T09:35:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008820c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-hz87cjqncdvb" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8e715185-33d5-494b-976c-25b2ab261cc3",
      "name": "8e715185-33d5-494b-976c-25b2ab261cc3",
      "status": "Failed",
      "startTime": "2026-09-16T09:35:58.536453378Z",
      "endTime": "2026-09-16T10:00:02.536562643Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1371bjtto4nf5m6o0pid4pjvle6qc still exists (deletion dispatched at 2026-09-16T09:35:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019ec138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gql2drdp7tjr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "name": "92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "status": "Failed",
      "startTime": "2026-09-16T09:34:43.828694597Z",
      "endTime": "2026-09-16T09:58:52.519202639Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1361u3o4ev8vfck5hk2a42dp30j3q still exists (deletion dispatched at 2026-09-16T09:34:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019ec138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gql2drdp7tjr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "name": "92c64afb-54ba-4f5e-a0ce-1a57e90c2b0b",
      "status": "Failed",
      "startTime": "2026-09-16T09:34:43.828694597Z",
      "endTime": "2026-09-16T09:58:52.519202639Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1361u3o4ev8vfck5hk2a42dp30j3q still exists (deletion dispatched at 2026-09-16T09:34:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892100141632635539456
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001580078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dfbjv854d6w5" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "name": "d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:31.257109777Z",
      "endTime": "2026-09-16T10:00:32.532474531Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1375769r7k6ubu137jlbqtq9j35it still exists (deletion dispatched at 2026-09-16T09:36:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001580078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-dfbjv854d6w5" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "name": "d99ec0f0-e1d1-41b0-b753-9ba0ef65c4b3",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:31.257109777Z",
      "endTime": "2026-09-16T10:00:32.532474531Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss1375769r7k6ubu137jlbqtq9j35it still exists (deletion dispatched at 2026-09-16T09:36:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011720a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-4cfzlz" in resourcegroup="rg-zstream-upgrade-5-0-2bmmvg6nm4m6" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "name": "a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "status": "Failed",
      "startTime": "2026-09-16T09:52:39.028633437Z",
      "endTime": "2026-09-16T10:16:42.563449333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss136411ov65aeqqn1ut2t4iprrujv9 still exists (deletion dispatched at 2026-09-16T09:52:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011720a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-4cfzlz" in resourcegroup="rg-zstream-upgrade-5-0-2bmmvg6nm4m6" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "name": "a49e79c0-b564-4703-ad68-3e8fbb555ee7",
      "status": "Failed",
      "startTime": "2026-09-16T09:52:39.028633437Z",
      "endTime": "2026-09-16T10:16:42.563449333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss136411ov65aeqqn1ut2t4iprrujv9 still exists (deletion dispatched at 2026-09-16T09:52:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010be048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lq45ld6h888d" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a",
      "name": "76270801-127d-4225-97dc-06e57322bf5a",
      "status": "Failed",
      "startTime": "2026-09-16T09:37:08.667361714Z",
      "endTime": "2026-09-16T10:01:12.766789046Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss13j30utlk9cm4u8n21tm903korv38 still exists (deletion dispatched at 2026-09-16T09:37:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010be048>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lq45ld6h888d" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76270801-127d-4225-97dc-06e57322bf5a",
      "name": "76270801-127d-4225-97dc-06e57322bf5a",
      "status": "Failed",
      "startTime": "2026-09-16T09:37:08.667361714Z",
      "endTime": "2026-09-16T10:01:12.766789046Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss13j30utlk9cm4u8n21tm903korv38 still exists (deletion dispatched at 2026-09-16T09:37:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892100141632635539456
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000aaaa60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6mp8vkxcbh5s" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "name": "d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "status": "Failed",
      "startTime": "2026-09-16T10:51:43.295429479Z",
      "endTime": "2026-09-16T11:20:49.710317459Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000aaaa60>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-6mp8vkxcbh5s" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "name": "d35dfb81-23c2-4a18-80a3-7d5bf5893728",
      "status": "Failed",
      "startTime": "2026-09-16T10:51:43.295429479Z",
      "endTime": "2026-09-16T11:20:49.710317459Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0011044c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-95l622wn4sm9" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "name": "1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.869364535Z",
      "endTime": "2026-09-16T11:21:09.719035401Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc0011044c0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-95l622wn4sm9" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "name": "1b24629a-f978-4a3d-9d8d-8d5b6776590f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.869364535Z",
      "endTime": "2026-09-16T11:21:09.719035401Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67912100161923835236352
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PR--------------------------------------------------------------------------------
full failure pattern:
--------------------------------------------------------------------------------
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-update-nodes: failed waiting for nodepool="np-update-nodes" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ckch4xg6nxh4" to finish creating: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84",
  "name": "e55eeddb-0ead-432e-a110-c696080abf84",
  "status": "Failed",
  "startTime": "2026-09-16T20:39:13.615216045Z",
  "endTime": "2026-09-16T20:58:16.977517435Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:131]: nodepool creation errors: [failed to create NodePool np-update-nodes: failed waiting for nodepool="np-update-nodes" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ckch4xg6nxh4" to finish creating: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e55eeddb-0ead-432e-a110-c696080abf84",
  "name": "e55eeddb-0ead-432e-a110-c696080abf84",
  "status": "Failed",
  "startTime": "2026-09-16T20:39:13.615216045Z",
  "endTime": "2026-09-16T20:58:16.977517435Z",
  "error": {
    "code": "InternalServerError",
    "message": "node pool creation did not complete before the deadline"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69452100313784206757888
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRAdmin credentials did not return 409 conflict error while cluster is deploying
full failure pattern:
Admin credentials did not return 409 conflict error while cluster is deploying
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:144]: Admin credentials did not return 409 conflict error while cluster is deploying
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:144]: Admin credentials did not return 409 conflict error while cluster is deploying
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67522100237146764873728
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code BadRequest; detail message The collection with name [Resources] in provided scope [/dbs/<cosmos-account>/colls/Resources] could not be fo...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code BadRequest; detail message The collection with name [Resources] in provided scope [/dbs/<cosmos-account>/colls/Resources] could not be found.\\\ \\ ActivityId: <uuid>, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0; provider Microsoft.DocumentDB
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
time=2026-09-16T13:31:49.803Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-09-16T13:31:51.360Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-09-16T13:31:52.622Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=6a654e03e0d8cd3ba2dcf4b7ca7ea7b9161da4e2734cccce092b3f0e303e2fe9 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6373376-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6a654e03e0d8cd3ba2dcf4b7ca7ea7b9161da4e2734cccce092b3f0e303e2fe9
time=2026-09-16T13:40:24.258Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6373376-svc/providers/Microsoft.Resources/deployments/6a654e03e0d8cd3ba2dcf4b7ca7ea7b9161da4e2734cccce092b3f0e303e2fe9/operationStatuses/08584120413735528505\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6373376/providers/Microsoft.Resources/deployments/rp_cosmos_db\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6373376/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j6373376/sqlRoleAssignments/3a943d25-7e89-52ef-bdd2-d3814561859c\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"BadRequest\\\",\\r\\n                \\\"message\\\": \\\"The collection with name [Resources] in provided scope [/dbs/arohcpci01-rp-j6373376/colls/Resources] could not be found.\\\\r\\\\nActivityId: 385582e2-ce0e-4904-a9f2-7469a3089fa3, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68862100209889606373376
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardEdsv6Family...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardEdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
time=2026-09-16T15:02:01.563Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T15:02:02.231Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T15:02:03.235Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=9b458a858b62ced8b861c50221db722ca7b84af5c0247e7866fc1bf068dbe8ac portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7202944-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9b458a858b62ced8b861c50221db722ca7b84af5c0247e7866fc1bf068dbe8ac
time=2026-09-16T15:11:05.045Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/9b458a858b62ced8b861c50221db722ca7b84af5c0247e7866fc1bf068dbe8ac/operationStatuses/08584120359627896142\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/cluster-oqca7h7ulqwlw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.Resources/deployments/user-agent-pools\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7202944-mgmt-2/agentPools/userswft3\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 96, remaining 48 for family StandardEdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7202944-mgmt-2/agentPools/userswft2\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 96, remaining 48 for family StandardEdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  },\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"BadRequest\\\",\\r\\n                    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7202944-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7202944-mgmt-2/agentPools/userswft1\\\",\\r\\n                    \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 96, remaining 48 for family StandardEdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69802100233505077202944
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00230a300>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ql5j7w6fx6vn" to finish creating: GET https://rp.j6364288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728",
      "name": "5fde05ba-68c2-437a-84ea-82765b356728",
      "status": "Failed",
      "startTime": "2026-09-16T11:04:09.351110934Z",
      "endTime": "2026-09-16T11:04:15.302518412Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00230a300>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ql5j7w6fx6vn" to finish creating: GET https://rp.j6364288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5fde05ba-68c2-437a-84ea-82765b356728",
      "name": "5fde05ba-68c2-437a-84ea-82765b356728",
      "status": "Failed",
      "startTime": "2026-09-16T11:04:09.351110934Z",
      "endTime": "2026-09-16T11:04:15.302518412Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #66882100164075236364288
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator, ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000edc320>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-d77hgbbbw6rf" to finish creating: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd",
      "name": "74e66356-25b3-4d6f-95af-f1292008b6fd",
      "status": "Failed",
      "startTime": "2026-09-16T20:47:22.69341748Z",
      "endTime": "2026-09-16T21:16:24.116227211Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 6m20s ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000edc320>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-d77hgbbbw6rf" to finish creating: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/74e66356-25b3-4d6f-95af-f1292008b6fd",
      "name": "74e66356-25b3-4d6f-95af-f1292008b6fd",
      "status": "Failed",
      "startTime": "2026-09-16T20:47:22.69341748Z",
      "endTime": "2026-09-16T21:16:24.116227211Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 6m20s ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69692100314659356676096
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-lzpm5g" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d707c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-lzpm5g: failed waiting for cluster="cp-ystream-upgrade-4-22-lzpm5g" in resourcegroup="rg-cp-ystream-upgrade-4-22-v5f9s54xdgwx" to finish creating: GET https://rp.j9719424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05",
      "name": "cff0006a-d905-424f-9efd-d8db75f2fd05",
      "status": "Failed",
      "startTime": "2026-09-16T20:49:09.853286386Z",
      "endTime": "2026-09-16T21:18:11.008528958Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 13m4s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-lzpm5g" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d707c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-lzpm5g: failed waiting for cluster="cp-ystream-upgrade-4-22-lzpm5g" in resourcegroup="rg-cp-ystream-upgrade-4-22-v5f9s54xdgwx" to finish creating: GET https://rp.j9719424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cff0006a-d905-424f-9efd-d8db75f2fd05",
      "name": "cff0006a-d905-424f-9efd-d8db75f2fd05",
      "status": "Failed",
      "startTime": "2026-09-16T20:49:09.853286386Z",
      "endTime": "2026-09-16T21:18:11.008528958Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 13m4s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #64032100309343839719424
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f92288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-djgr48qrs9vr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "name": "a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:21.892022588Z",
      "endTime": "2026-09-16T10:00:22.50023585Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12t13cvqgvb69duuok3m0c1if4g29 still exists (deletion dispatched at 2026-09-16T09:36:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f92288>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-djgr48qrs9vr" to finish deleting: GET https://rp.j5539456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "name": "a17df9ed-a5b1-4a4c-b59d-0c40cfdd60e7",
      "status": "Failed",
      "startTime": "2026-09-16T09:36:21.892022588Z",
      "endTime": "2026-09-16T10:00:22.50023585Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss12t13cvqgvb69duuok3m0c1if4g29 still exists (deletion dispatched at 2026-09-16T09:36:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892100141632635539456
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b4030>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-sk5lk2xqrw7v" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e",
      "name": "234e26dc-5c76-4335-9749-4a1c49aae81e",
      "status": "Failed",
      "startTime": "2026-09-16T17:00:40.136241522Z",
      "endTime": "2026-09-16T17:24:42.83613595Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7is141l1pe3gtmk90b2cej7ltp3gn still exists (deletion dispatched at 2026-09-16T17:00:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012b4030>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-sk5lk2xqrw7v" to finish deleting: GET https://rp.j4807552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/234e26dc-5c76-4335-9749-4a1c49aae81e",
      "name": "234e26dc-5c76-4335-9749-4a1c49aae81e",
      "status": "Failed",
      "startTime": "2026-09-16T17:00:40.136241522Z",
      "endTime": "2026-09-16T17:24:42.83613595Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ss7is141l1pe3gtmk90b2cej7ltp3gn still exists (deletion dispatched at 2026-09-16T17:00:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892100252330694807552
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00108c078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mzfrqt8tf9wq" to finish deleting: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "name": "e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "status": "Failed",
      "startTime": "2026-09-16T11:18:42.744808437Z",
      "endTime": "2026-09-16T11:42:49.617012035Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00108c078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mzfrqt8tf9wq" to finish deleting: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "name": "e9f01e6a-fcf9-4bbb-aeeb-61ee6e0ac502",
      "status": "Failed",
      "startTime": "2026-09-16T11:18:42.744808437Z",
      "endTime": "2026-09-16T11:42:49.617012035Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67912100161923835236352
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] BackendControllerRetryHotLoop fired
full failure pattern:
alert [svc] BackendControllerRetryHotLoop fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T15:28:20Z
  Ended:       2026-09-16T15:40:21Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerRetryHotLoop", cluster="ci01-j8680064-svc", component="backend", name="dataplaneoidcfederation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue dataplaneoidcfederation has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerRetryHotLoop does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69832100226936398680064
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] KubeContainerWaiting fired
full failure pattern:
alert [svc] KubeContainerWaiting fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="secrets-store", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-cwxxr in namespace kube-system on container secrets-store has been in waiting state for longer than 1 hour.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="liveness-probe", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-cwxxr in namespace kube-system on container liveness-probe has been in waiting state for longer than 1 hour.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="node-driver-registrar", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: pod/aks-secrets-store-csi-driver-cwxxr in namespace kube-system on container node-driver-registrar has been in waiting state for longer than 1 hour.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="prometheus-collector", namespace="kube-system", pod="ama-metrics-node-xfmjp", region="canadacentral", severity="warning"
  Description: pod/ama-metrics-node-xfmjp in namespace kube-system on container prometheus-collector has been in waiting state for longer than 1 hour.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:30:16Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerWaiting", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="addon-token-adapter", namespace="kube-system", pod="ama-metrics-node-xfmjp", region="canadacentral", severity="warning"
  Description: pod/ama-metrics-node-xfmjp in namespace kube-system on container addon-token-adapter has been in waiting state for longer than 1 hour.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeContainerWaiting does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69822100248871216091136
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] KubeNodeNotReady fired
full failure pattern:
alert [svc] KubeNodeNotReady fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:00:36Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="prometheus", node="aks-system-28181016-vmss000000", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-system-28181016-vmss000000 has been unready for more than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:00:36Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", condition="Ready", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="prometheus", node="aks-system-28181016-vmss000000", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="true"
  Description: aks-system-28181016-vmss000000 has been unready for more than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeNodeNotReady does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69822100248871216091136
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] LeaderElectionLeaseStale fired
full failure pattern:
alert [svc] LeaderElectionLeaseStale fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:54:24Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j4807552-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.89:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4807552/providers/microsoft.monitor/accounts/services-j4807552", namespace="klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn", pod="arohcp-monitor-kube-state-metrics-574f657d96-zvqnn", prometheus="prometheus/prometheus", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn on cluster ci01-j4807552-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:54:24Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j4807552-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.89:8080", job="kube-state-metrics", lease="config-policy-controller", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4807552/providers/microsoft.monitor/accounts/services-j4807552", namespace="klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn", pod="arohcp-monitor-kube-state-metrics-574f657d96-zvqnn", prometheus="prometheus/prometheus", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease config-policy-controller in namespace klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn on cluster ci01-j4807552-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:54:24Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j4807552-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.89:8080", job="kube-state-metrics", lease="governance-policy-framework", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4807552/providers/microsoft.monitor/accounts/services-j4807552", namespace="klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn", pod="arohcp-monitor-kube-state-metrics-574f657d96-zvqnn", prometheus="prometheus/prometheus", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease governance-policy-framework in namespace klusterlet-2ss7is141l1pe3gtmk90b2cej7ltp3gn on cluster ci01-j4807552-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892100252330694807552
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] userJourneyFrontendLatencyP991h5m fired
full failure pattern:
alert [svc] userJourneyFrontendLatencyP991h5m fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:39:29Z
  Ended:       2026-09-16T09:50:27Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP991h5m", cluster="ci01-j3290112-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3290112/providers/microsoft.monitor/accounts/services-j3290112", severity="3", short_window="5m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j3290112-svc is above 5s on both the 1h and 5m windows (severe miss of the p99 < 1s ARM sync SLO).
Firing 2:
  State:       Resolved
  Started:     2026-09-16T10:10:33Z
  Ended:       2026-09-16T10:17:29Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP991h5m", cluster="ci01-j3290112-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3290112/providers/microsoft.monitor/accounts/services-j3290112", severity="3", short_window="5m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j3290112-svc is above 5s on both the 1h and 5m windows (severe miss of the p99 < 1s ARM sync SLO).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert userJourneyFrontendLatencyP991h5m does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #65032100132038643290112
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRbreakglass session kubeconfig not ready: timeout waiting for session to become ready; HostedControlPlane exists but is not ready
full failure pattern:
breakglass session kubeconfig not ready: timeout waiting for session to become ready; HostedControlPlane exists but is not ready
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:185]: failed to create SRE breakglass credentials with aro-sre-pso permissions
Unexpected error:
    <*fmt.wrapError | 0xc000a7f820>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-wlfqkcdjh9rm/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-lbdxk/kubeconfig: timeout waiting for session to become ready (last status: {"HostedControlPlaneAvailable":{"message":"HostedControlPlane exists but is not ready","reason":"HostedControlPlaneNotReady","status":"False"},"status":"Session is not ready"})
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:185]: failed to create SRE breakglass credentials with aro-sre-pso permissions
Unexpected error:
    <*fmt.wrapError | 0xc000a7f820>: 
    failed to get ready session kubeconfig from /admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/admin-api-breakglass-wlfqkcdjh9rm/providers/microsoft.redhatopenshift/hcpopenshiftclusters/sre-hcp-cluster/breakglass/breakglass-lbdxk/kubeconfig: timeout waiting for session to become ready (last status: {"HostedControlPlaneAvailable":{"message":"HostedControlPlane exists but is not ready","reason":"HostedControlPlaneNotReady","status":"False"},"status":"Session is not ready"})
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #63642100153419804184576
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRclusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no regis...
full failure pattern:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "0IciLMVmLbqimBoU8f5q6QBEpmWjuJNgDzWaCM2v9MM="
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001ffe8a0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "0IciLMVmLbqimBoU8f5q6QBEpmWjuJNgDzWaCM2v9MM="
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001ffe8a0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "0IciLMVmLbqimBoU8f5q6QBEpmWjuJNgDzWaCM2v9MM="
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67912100252538870697984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0011eef40>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gn4q6q4pqf7b" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406",
      "name": "c302cb08-5816-4f3c-a786-daa45a7d6406",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:22.418838081Z",
      "endTime": "2026-09-16T15:13:01.450563178Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0011eef40>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gn4q6q4pqf7b" to finish creating: GET https://rp.j4331648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c302cb08-5816-4f3c-a786-daa45a7d6406",
      "name": "c302cb08-5816-4f3c-a786-daa45a7d6406",
      "status": "Failed",
      "startTime": "2026-09-16T15:12:22.418838081Z",
      "endTime": "2026-09-16T15:13:01.450563178Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69762100222840434331648
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/18 context deadline exceeded
full failure pattern:
failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/18 context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
time=2026-09-16T09:03:26.822Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 description="Step kubelet-ds\n Kind: Helm\n"
time=2026-09-16T09:03:27.157Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 values="map[enabled:true kubeletParametersImage:mcr.microsoft.com/aks/command/runtime@sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23]"
time=2026-09-16T09:03:27.157Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-09-16T09:03:27.284Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 namespace=kubelet-ds
time=2026-09-16T09:03:27.475Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 dryRun=false
time=2026-09-16T09:03:27.475Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-09-16T09:03:27.530Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2
time=2026-09-16T09:08:28.583Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/18\ncontext deadline exceeded"
time=2026-09-16T09:08:28.583Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/18\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/kubelet-ds (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #60092100132003708932096
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRtimeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.13%1 of 751 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0009db7a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wngrxcmwz6zh" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901AndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-wngrxcmwz6zh, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0009db7a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wngrxcmwz6zh" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20260901AndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-wngrxcmwz6zh, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #63642100229912697769984
alert [svc] ServiceCPUDrift fired
full failure pattern:
alert [svc] ServiceCPUDrift fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert466.13%46 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 46none
Full failure examples (3)
alert fired 12 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:34:22Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kv8ld", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-kv8ld (namespace arobit) on cluster ci01-j7494016-svc has averaged 177.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:32:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b8f98bb8d-xxtkh", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b8f98bb8d-xxtkh (namespace mgmt-agent) on cluster ci01-j7494016-mgmt-1 has averaged 189.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:40:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8987ccb77-nmgwd", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8987ccb77-nmgwd (namespace clusters-service) on cluster ci01-j7494016-svc has averaged 256.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:29:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-7b8f98bb8d-l5m85", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-7b8f98bb8d-l5m85 (namespace mgmt-agent) on cluster ci01-j7494016-mgmt-2 has averaged 180% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:33:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-hmnhn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-hmnhn (namespace arobit) on cluster ci01-j7494016-mgmt-1 has averaged 217.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:29:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-d9thl", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-d9thl (namespace arobit) on cluster ci01-j7494016-mgmt-2 has averaged 167.9% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:14:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-85694f5cf9-96x49", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-85694f5cf9-96x49 (namespace aro-hcp) on cluster ci01-j7494016-svc has averaged 418.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 8:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:40:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8987ccb77-btgnn", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8987ccb77-btgnn (namespace clusters-service) on cluster ci01-j7494016-svc has averaged 253.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 9:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:30:19Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-w55jk", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-w55jk (namespace arobit) on cluster ci01-j7494016-mgmt-1 has averaged 147.3% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 10:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:27:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-bwg62", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-bwg62 (namespace arobit) on cluster ci01-j7494016-svc has averaged 130.7% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-17T02:14:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-8987ccb77-sfcv9", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-8987ccb77-sfcv9 (namespace clusters-service) on cluster ci01-j7494016-svc has averaged 850.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 12:
  State:       Resolved
  Started:     2026-09-17T02:14:20Z
  Ended:       2026-09-17T02:33:21Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7494016-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-v5ccx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-v5ccx (namespace arobit) on cluster ci01-j7494016-mgmt-1 has averaged 160.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T00:07:06Z
  Ended:       2026-09-17T00:23:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7559936-mgmt-2", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-554b597df9-2phx7", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-554b597df9-2phx7 (namespace mgmt-agent) on cluster ci01-j7559936-mgmt-2 has averaged 182% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T00:07:06Z
  Ended:       2026-09-17T00:26:06Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j7559936-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7ddmw", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-7ddmw (namespace arobit) on cluster ci01-j7559936-mgmt-2 has averaged 200.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.
alert fired 6 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:10:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-mgmt-1", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2pvnx", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2pvnx (namespace arobit) on cluster ci01-j3332864-mgmt-1 has averaged 186.4% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:10:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-mgmt-1", component="service-memory", container="mgmt-agent-controller", namespace="mgmt-agent", pod="mgmt-agent-dd7947c67-r26c4", severity="warning", team="hcp-sl"
  Description: Container mgmt-agent-controller in pod mgmt-agent-dd7947c67-r26c4 (namespace mgmt-agent) on cluster ci01-j3332864-mgmt-1 has averaged 191.2% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 3:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:13:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-dd85c69cd-czx9t", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-dd85c69cd-czx9t (namespace clusters-service) on cluster ci01-j3332864-svc has averaged 238.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 4:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:13:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="aro-hcp-backend", namespace="aro-hcp", pod="aro-hcp-backend-558d75db97-9xbgb", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-backend in pod aro-hcp-backend-558d75db97-9xbgb (namespace aro-hcp) on cluster ci01-j3332864-svc has averaged 416.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 5:
  State:       Resolved
  Started:     2026-09-17T00:03:50Z
  Ended:       2026-09-17T00:13:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-72f4l", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-72f4l (namespace arobit) on cluster ci01-j3332864-svc has averaged 179.5% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-17T00:28:51Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j3332864-svc", component="service-memory", container="clusters-service-server", namespace="clusters-service", pod="clusters-service-dd85c69cd-vfcn8", severity="warning", team="hcp-sl"
  Description: Container clusters-service-server in pod clusters-service-dd85c69cd-vfcn8 (namespace clusters-service) on cluster ci01-j3332864-svc has averaged 230.1% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ServiceCPUDrift does not fire46
Affected runs (46)
Date (UTC)Associated PRProw job
PR #69772100373409937494016
PR #69402100354655547559936
PR #69402100353767223332864
PR #68942100351996258160640
PR #68802100351264033345536
PR #68942100351223038218240
PR #48782100346022097063936
PR #69452100344321164185600
PR #69742100343556559343616
PR #69772100335879556960256
PR #68562100335557920952320
PR #67522100331448274784256
PR #69802100328454825185280
PR #69662100327783220645888
PR #63642100324480323686400
PR #68052100323463913803776
PR #69692100314659356676096
PR #69452100313784206757888
PR #69042100310890917138432
PR #68942100309676229922816
PR #64032100309343839719424
PR #69282100309325879709696
PR #48782100309243423887360
PR #67522100299874200195072
PR #69662100298749917007872
PR #69762100298527354654720
PR #69772100298495381475328
PR #69802100294017282478080
PR #68052100293442402783232
PR #63642100285588732121088
PR #69452100280971763912704
PR #68942100273649465430016
PR #69692100268367708426240
PR #69662100263412582846464
PR #69282100262131709513728
PR #69772100260298874163200
PR #68562100260058020450304
PR #69762100253677548736512
PR #69802100253157446651904
PR #67912100252538870697984
PR #69642100252533153861632
PR #68892100252330694807552
PR #69822100248871216091136
PR #69232100244030364323840
PR #69232100243112008880128
PR #67262100185438990897152
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert303.99%30 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 30none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:46:37Z
  Ended:       2026-09-17T00:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7559936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdiuonvecl8bulup4napa18plli3hl-v5g6h1u7r3y6b9q", pod="router-77b5d8dffb-95c5m", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdiuonvecl8bulup4napa18plli3hl-v5g6h1u7r3y6b9q/router-77b5d8dffb-95c5m has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T23:46:37Z
  Ended:       2026-09-17T00:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7559936-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdisk0mggfhm9130ika04epdikqlho-s5n1l0v6o6a7b8a", pod="router-55df5c6c57-b4j8r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdisk0mggfhm9130ika04epdikqlho-s5n1l0v6o6a7b8a/router-55df5c6c57-b4j8r has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:38:16Z
  Ended:       2026-09-17T00:05:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8160640-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdff0k383bfd7grp5qnbnt5v4r1b0u-d3b2r2a4t0o4l6a", pod="router-5564cd696f-nb8mv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdff0k383bfd7grp5qnbnt5v4r1b0u-d3b2r2a4t0o4l6a/router-5564cd696f-nb8mv has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T23:31:05Z
  Ended:       2026-09-16T23:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3345536-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2ssdc08lq86mikr3m7qtfj8ulr4bs703-h7y5g4s0a4a5j6c", pod="router-5bb8b7c8bd-thvsf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2ssdc08lq86mikr3m7qtfj8ulr4bs703-h7y5g4s0a4a5j6c/router-5bb8b7c8bd-thvsf has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire30
Affected runs (30)
Date (UTC)Associated PRProw job
PR #69402100354655547559936
PR #68942100351996258160640
PR #68802100351264033345536
PR #69452100344321164185600
PR #69772100335879556960256
PR #69662100327783220645888
PR #63642100324480323686400
PR #68052100323463913803776
PR #69452100313784206757888
PR #69662100298749917007872
PR #69772100298495381475328
PR #69692100268367708426240
PR #69282100262131709513728
PR #68802100231086113034240
PR #63642100229912697769984
PR #69832100226936398680064
PR #69762100222840434331648
PR #69682100213017273700352
PR #68982100207889619619840
PR #66722100194205623128064
PR #69762100189557025673216
PR #69172100189503086923776
PR #69522100176649516486656
PR #69222100161384225443840
PR #63642100153419804184576
PR #68892100141632635539456
PR #68792100132080049459200
PR #69622100132023019507712
PR #66722100129038159843328
PR #68562100124639266082816
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e273.60%27 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 71none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fcc7a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-jbw245668lwj" to finish creating: GET https://rp.j7559936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f",
      "name": "6d2a250e-0bac-4f44-a708-541cb00c701f",
      "status": "Failed",
      "startTime": "2026-09-16T23:34:27.803730917Z",
      "endTime": "2026-09-17T00:03:30.376925674Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fcc7a0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-jbw245668lwj" to finish creating: GET https://rp.j7559936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6d2a250e-0bac-4f44-a708-541cb00c701f",
      "name": "6d2a250e-0bac-4f44-a708-541cb00c701f",
      "status": "Failed",
      "startTime": "2026-09-16T23:34:27.803730917Z",
      "endTime": "2026-09-17T00:03:30.376925674Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-v9lkhvnzdhld/cluster-candidate-4-21-jrkc8h should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ba8a40>: 
    failed to create HCP cluster cluster-candidate-4-21-jrkc8h: failed waiting for cluster="cluster-candidate-4-21-jrkc8h" in resourcegroup="rg-candidate-4-21-v9lkhvnzdhld" to finish creating: GET https://rp.j8160640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38",
      "name": "61166157-eac2-4677-a0c1-58bf48e34b38",
      "status": "Failed",
      "startTime": "2026-09-16T23:26:49.985549857Z",
      "endTime": "2026-09-16T23:55:49.56140206Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-v9lkhvnzdhld/cluster-candidate-4-21-jrkc8h should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ba8a40>: 
    failed to create HCP cluster cluster-candidate-4-21-jrkc8h: failed waiting for cluster="cluster-candidate-4-21-jrkc8h" in resourcegroup="rg-candidate-4-21-v9lkhvnzdhld" to finish creating: GET https://rp.j8160640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/61166157-eac2-4677-a0c1-58bf48e34b38",
      "name": "61166157-eac2-4677-a0c1-58bf48e34b38",
      "status": "Failed",
      "startTime": "2026-09-16T23:26:49.985549857Z",
      "endTime": "2026-09-16T23:55:49.56140206Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f12a00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mt67hxmh7jr7" to finish creating: GET https://rp.j7063936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "name": "f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "status": "Failed",
      "startTime": "2026-09-16T22:42:15.998420325Z",
      "endTime": "2026-09-16T23:11:18.627710941Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f12a00>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mt67hxmh7jr7" to finish creating: GET https://rp.j7063936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "name": "f2d6116a-8dc5-497c-a1b9-37dbf3d9da56",
      "status": "Failed",
      "startTime": "2026-09-16T22:42:15.998420325Z",
      "endTime": "2026-09-16T23:11:18.627710941Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 19m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (26)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE can pause schedules to stop backup execution for an HCP cluster7
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.226
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.215
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property5
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (27)
Date (UTC)Associated PRProw job
PR #69402100354655547559936
PR #68942100351996258160640
PR #48782100346022097063936
PR #69452100344321164185600
PR #63642100324480323686400
PR #68052100323463913803776
PR #67522100299874200195072
PR #69802100294017282478080
PR #69692100268367708426240
PR #69282100262131709513728
PR #69642100252533153861632
PR #68802100231086113034240
PR #63642100229912697769984
PR #68562100228663373991936
PR #64392100227428122103808
PR #69832100226936398680064
PR #69762100222840434331648
PR #69682100213017273700352
PR #67912100196783199096832
PR #69762100189557025673216
PR #69172100189503086923776
PR #69222100161384225443840
PR #63642100153419804184576
PR #58462100147953778823168
PR #68892100141632635539456
PR #68792100132080049459200
PR #69392100119365197238272
failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded
full failure pattern:
failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision101.33%10 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 10none
Full failure examples (3)
time=2026-09-16T21:39:23.882Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T21:39:24.651Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-16T21:39:25.419Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=1957e733d563fe8d01011d2966a166813e9314307467a323647f090022c9d7a7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1475712-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F1957e733d563fe8d01011d2966a166813e9314307467a323647f090022c9d7a7
time=2026-09-16T22:09:23.556Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
time=2026-09-16T17:32:16.355Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T17:32:17.069Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-16T17:32:18.139Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=2220b5a771141f6ca7f14ed4462ae3b3d70e20f12f11d2768e401b8114aa657f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7836672-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F2220b5a771141f6ca7f14ed4462ae3b3d70e20f12f11d2768e401b8114aa657f
time=2026-09-16T18:02:16.065Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
time=2026-09-16T14:44:36.842Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-16T14:44:37.777Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-16T14:44:38.583Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=00e0f6eaf1a87ba23e25957419627b2f2bf902c90c40b4a9a0680ec1b94026a9 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1035008-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F00e0f6eaf1a87ba23e25957419627b2f2bf902c90c40b4a9a0680ec1b94026a9
time=2026-09-16T15:14:36.624Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=2)6
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster (stamp=1)4
Affected runs (10)
Date (UTC)Associated PRProw job
PR #68202100332359231475712
PR #68202100270038647836672
PR #69772100228886901035008
PR #67792100198040831791104
PR #64392100194885893099520
PR #69772100194111955931136
PR #58462100192526756483072
PR #69742100190163022909440
PR #66942100169554075324416
PR #69722100119215724826624
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e81.07%8 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 22none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001400078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-dqgbxbhrnvkj" to finish deleting: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b",
      "name": "38120612-3f62-4d83-b1c9-f450e308488b",
      "status": "Failed",
      "startTime": "2026-09-16T20:57:03.974907545Z",
      "endTime": "2026-09-16T21:21:04.12131988Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssb46njkre58n4p9mol1i33anuaqjtv still exists (deletion dispatched at 2026-09-16T20:57:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001400078>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-dqgbxbhrnvkj" to finish deleting: GET https://rp.j6676096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/38120612-3f62-4d83-b1c9-f450e308488b",
      "name": "38120612-3f62-4d83-b1c9-f450e308488b",
      "status": "Failed",
      "startTime": "2026-09-16T20:57:03.974907545Z",
      "endTime": "2026-09-16T21:21:04.12131988Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssb46njkre58n4p9mol1i33anuaqjtv still exists (deletion dispatched at 2026-09-16T20:57:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008e6240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-x4ht6zckdvtk" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab",
      "name": "675795c0-1d48-4443-bf17-96b820fb9dab",
      "status": "Failed",
      "startTime": "2026-09-16T20:56:31.8857491Z",
      "endTime": "2026-09-16T21:20:37.108128013Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssari2k27ic1890mj0nf8obfposs9em still exists (deletion dispatched at 2026-09-16T20:56:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0008e6240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-x4ht6zckdvtk" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/675795c0-1d48-4443-bf17-96b820fb9dab",
      "name": "675795c0-1d48-4443-bf17-96b820fb9dab",
      "status": "Failed",
      "startTime": "2026-09-16T20:56:31.8857491Z",
      "endTime": "2026-09-16T21:20:37.108128013Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssari2k27ic1890mj0nf8obfposs9em still exists (deletion dispatched at 2026-09-16T20:56:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001910318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-tfkgg85wrhjp" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51",
      "name": "40b6c313-a567-4259-bb1a-298290750f51",
      "status": "Failed",
      "startTime": "2026-09-16T20:55:14.718059601Z",
      "endTime": "2026-09-16T21:19:17.098694741Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssarn8fr2snie5k70mjfefs0cg1miti still exists (deletion dispatched at 2026-09-16T20:55:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001910318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-tfkgg85wrhjp" to finish deleting: GET https://rp.j6757888.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/40b6c313-a567-4259-bb1a-298290750f51",
      "name": "40b6c313-a567-4259-bb1a-298290750f51",
      "status": "Failed",
      "startTime": "2026-09-16T20:55:14.718059601Z",
      "endTime": "2026-09-16T21:19:17.098694741Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ssarn8fr2snie5k70mjfefs0cg1miti still exists (deletion dispatched at 2026-09-16T20:55:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (16)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20260901preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (8)
Date (UTC)Associated PRProw job
PR #69692100314659356676096
PR #69452100313784206757888
PR #68892100252330694807552
PR #63642100229912697769984
PR #69832100226936398680064
PR #69522100222891458039808
PR #69762100222840434331648
PR #63642100188464501755904
alert [svc] FrontendPathMedianLatency fired
full failure pattern:
alert [svc] FrontendPathMedianLatency fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert81.07%8 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 8none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:39:57Z
  Ended:       2026-09-16T17:45:56Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0450304-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0450304-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:27:30Z
  Ended:       2026-09-16T17:34:29Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3861632-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3861632-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T16:39:46Z
  Ended:       2026-09-16T17:12:43Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6091136-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6091136-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire8
Affected runs (8)
Date (UTC)Associated PRProw job
PR #68562100260058020450304
PR #69642100252533153861632
PR #69822100248871216091136
PR #68562100228663373991936
PR #68792100132080049459200
PR #65032100132038643290112
PR #69232100110113200148480
PR #68942100104517457547264
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e70.93%7 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011165a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2nsvjcl7j6m6" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0",
      "name": "a27b380d-13cc-4abe-9932-546cf6483df0",
      "status": "Failed",
      "startTime": "2026-09-16T21:08:44.335887928Z",
      "endTime": "2026-09-16T21:37:44.000335858Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0011165a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-2nsvjcl7j6m6" to finish creating: GET https://rp.j3803776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a27b380d-13cc-4abe-9932-546cf6483df0",
      "name": "a27b380d-13cc-4abe-9932-546cf6483df0",
      "status": "Failed",
      "startTime": "2026-09-16T21:08:44.335887928Z",
      "endTime": "2026-09-16T21:37:44.000335858Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000db98e0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-5kbxx5fn4nnf" to finish creating: GET https://rp.j2783232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "name": "ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "status": "Failed",
      "startTime": "2026-09-16T19:26:24.241613754Z",
      "endTime": "2026-09-16T19:55:27.054305259Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000db98e0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-5kbxx5fn4nnf" to finish creating: GET https://rp.j2783232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "name": "ff402ee6-ae9a-437a-b814-5ad8ed11cf52",
      "status": "Failed",
      "startTime": "2026-09-16T19:26:24.241613754Z",
      "endTime": "2026-09-16T19:55:27.054305259Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vlzgbk with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc0006bee60>: 
    failed to create HCP cluster np-version-upgrade-cluster-vlzgbk: failed waiting for cluster="np-version-upgrade-cluster-vlzgbk" in resourcegroup="rg-np-version-upgrade-vlzgbk-6gtpz2tlsds2" to finish creating: GET https://rp.j2846464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691",
      "name": "83de4d68-f488-460e-8446-69209da0d691",
      "status": "Failed",
      "startTime": "2026-09-16T17:51:23.682356624Z",
      "endTime": "2026-09-16T18:20:27.523942323Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vlzgbk with version 4.21.33
Unexpected error:
    <*fmt.wrapError | 0xc0006bee60>: 
    failed to create HCP cluster np-version-upgrade-cluster-vlzgbk: failed waiting for cluster="np-version-upgrade-cluster-vlzgbk" in resourcegroup="rg-np-version-upgrade-vlzgbk-6gtpz2tlsds2" to finish creating: GET https://rp.j2846464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/83de4d68-f488-460e-8446-69209da0d691",
      "name": "83de4d68-f488-460e-8446-69209da0d691",
      "status": "Failed",
      "startTime": "2026-09-16T17:51:23.682356624Z",
      "endTime": "2026-09-16T18:20:27.523942323Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (7)
Date (UTC)Associated PRProw job
PR #68052100323463913803776
PR #68052100293442402783232
PR #69662100263412582846464
PR #69822100248871216091136
PR #68982100207889619619840
PR #67262100185438990897152
PR #69642100132115432607744
alert [svc] userJourneyFrontendLatencyP996h30m fired
full failure pattern:
alert [svc] userJourneyFrontendLatencyP996h30m fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert60.80%6 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 6none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T09:29:21Z
  Ended:       2026-09-16T09:49:20Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j5155200-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155200/providers/microsoft.monitor/accounts/services-j5155200", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j5155200-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T09:46:29Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j3290112-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3290112/providers/microsoft.monitor/accounts/services-j3290112", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j3290112-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T08:53:50Z
  Ended:       2026-09-16T09:06:52Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatencyP996h30m", cluster="ci01-j9843328-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9843328/providers/microsoft.monitor/accounts/services-j9843328", severity="3", short_window="30m", slo="frontend-latency"
  Description: Frontend p99 latency on cluster ci01-j9843328-svc is above 1s on both the 6h and 30m windows (p99 < 1s ARM sync SLO miss).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert userJourneyFrontendLatencyP996h30m does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #69332100132075855155200
PR #65032100132038643290112
PR #66722100129038159843328
PR #69712100127403039789056
PR #69232100110113200148480
PR #68942100104517457547264
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e60.80%6 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 6INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000bb5820>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xxwwr4bjs848" to finish creating: GET https://rp.j8426240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "name": "13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "status": "Failed",
      "startTime": "2026-09-16T18:00:53.783577799Z",
      "endTime": "2026-09-16T18:19:53.25990245Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000bb5820>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xxwwr4bjs848" to finish creating: GET https://rp.j8426240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "name": "13d2af80-0cd5-4fee-b2fe-b55a2b2248e3",
      "status": "Failed",
      "startTime": "2026-09-16T18:00:53.783577799Z",
      "endTime": "2026-09-16T18:19:53.25990245Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0008cd0a0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jvwfr7" in resourcegroup="rg-np-skip-minor-jvwfr7-7qvvgr9wwfbq" to finish creating: GET https://rp.j2424320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "name": "6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "status": "Failed",
      "startTime": "2026-09-16T14:13:13.201859753Z",
      "endTime": "2026-09-16T14:32:13.373200335Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0008cd0a0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-jvwfr7" in resourcegroup="rg-np-skip-minor-jvwfr7-7qvvgr9wwfbq" to finish creating: GET https://rp.j2424320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "name": "6c6ba019-61f3-4ad5-922c-02cdd95f824e",
      "status": "Failed",
      "startTime": "2026-09-16T14:13:13.201859753Z",
      "endTime": "2026-09-16T14:32:13.373200335Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001ff26c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-g5s8ftp9hh9t" to finish creating: GET https://rp.j4241024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "name": "06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "status": "Failed",
      "startTime": "2026-09-16T12:18:53.259035342Z",
      "endTime": "2026-09-16T12:37:53.675934016Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001ff26c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-g5s8ftp9hh9t" to finish creating: GET https://rp.j4241024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "name": "06ef0feb-5e7e-4993-bd90-39657ca016bf",
      "status": "Failed",
      "startTime": "2026-09-16T12:18:53.259035342Z",
      "endTime": "2026-09-16T12:37:53.675934016Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (6)
Date (UTC)Associated PRProw job
PR #69692100268367708426240
PR #68942100212360202424320
PR #69522100177539514241024
PR #68862100170313131102208
PR #66882100164075236364288
PR #67912100161923835236352
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert60.80%6 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 6none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T21:44:54Z
  Ended:       2026-09-16T21:51:52Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3803776-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.109:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3803776/providers/microsoft.monitor/accounts/hcps-j3803776", namespace="ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r", pod="arohcp-monitor-kube-state-metrics-574f657d96-gndsj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T21:45:52Z
  Ended:       2026-09-16T22:01:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3803776-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.109:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3803776/providers/microsoft.monitor/accounts/hcps-j3803776", namespace="ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r", pod="arohcp-monitor-kube-state-metrics-574f657d96-gndsj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ssben4sp2jl4ug8rvisqeb9vbtmfdbc-n8l2c7a7y0n1i2r/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T17:11:51Z
  Ended:       2026-09-16T17:29:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/hcps-j6091136", namespace="ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:30:00Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7769984/providers/microsoft.monitor/accounts/hcps-j7769984", namespace="ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-n4qlf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:30:00Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7769984-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.76:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7769984/providers/microsoft.monitor/accounts/hcps-j7769984", namespace="ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-574f657d96-n4qlf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2ss6631b7kcrqufnkep1rdniihr6hk1s-cilium-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #68052100323463913803776
PR #69822100248871216091136
PR #63642100229912697769984
PR #68982100207889619619840
PR #68892100141632635539456
PR #66722100129038159843328
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert50.67%5 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 5none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T18:21:59Z
  Ended:       2026-09-16T18:42:00Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2846464-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2846464/providers/microsoft.monitor/accounts/hcps-j2846464", namespace="ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t", pod="arohcp-monitor-kube-state-metrics-574f657d96-fbvzq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T18:21:59Z
  Ended:       2026-09-16T18:29:59Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2846464-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.181:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2846464/providers/microsoft.monitor/accounts/hcps-j2846464", namespace="ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t", pod="arohcp-monitor-kube-state-metrics-574f657d96-fbvzq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss8i74sks91slsm7oliign0ld9qcsch-q0f6x6b6a5y9c2t/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T16:56:51Z
  Ended:       2026-09-16T17:24:50Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/hcps-j6091136", namespace="ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss7f44euk2o4nqkrp28r72hngjs9t5m-g7f9d6m4t8e6w1m/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T14:02:53Z
  Ended:       2026-09-16T14:27:49Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9619840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9619840/providers/microsoft.monitor/accounts/hcps-j9619840", namespace="ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r", pod="arohcp-monitor-kube-state-metrics-574f657d96-q22d9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T14:02:53Z
  Ended:       2026-09-16T14:22:48Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9619840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9619840/providers/microsoft.monitor/accounts/hcps-j9619840", namespace="ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r", pod="arohcp-monitor-kube-state-metrics-574f657d96-q22d9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2ss4t4fdubst8s5b8tltfi3bmb1s95er-u3f5u3d4c4j0m6r/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #69662100263412582846464
PR #69822100248871216091136
PR #68982100207889619619840
PR #67262100185438990897152
PR #69642100132115432607744
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor ...
full failure pattern:
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e50.67%5 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000a2c540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-86c2sh" in resourcegroup="rg-np-dg-minor-86c2sh-42qzgk6zdhqx" to finish creating: GET https://rp.j4654720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0",
      "name": "b168b436-e53c-4032-86e5-822b4fbf8df0",
      "status": "Failed",
      "startTime": "2026-09-16T19:54:19.817293417Z",
      "endTime": "2026-09-16T19:54:53.786912449Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000a2c540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-86c2sh" in resourcegroup="rg-np-dg-minor-86c2sh-42qzgk6zdhqx" to finish creating: GET https://rp.j4654720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b168b436-e53c-4032-86e5-822b4fbf8df0",
      "name": "b168b436-e53c-4032-86e5-822b4fbf8df0",
      "status": "Failed",
      "startTime": "2026-09-16T19:54:19.817293417Z",
      "endTime": "2026-09-16T19:54:53.786912449Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002420258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-hgxdxgb7d5pf" to finish creating: GET https://rp.j3912704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "name": "bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "status": "Failed",
      "startTime": "2026-09-16T18:52:16.312207682Z",
      "endTime": "2026-09-16T18:52:49.366400193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc002420258>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-hgxdxgb7d5pf" to finish creating: GET https://rp.j3912704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "name": "bc0b2672-7030-4135-9a4f-3ac73ff9d19d",
      "status": "Failed",
      "startTime": "2026-09-16T18:52:16.312207682Z",
      "endTime": "2026-09-16T18:52:49.366400193Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv"
Unexpected error:
    <*fmt.wrapError | 0xc000d846a0>: 
    failed to create NodePool np-96ddmv: failed waiting for nodepool="np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv" in resourcegroup="rg-candidate-5-0-dxgjw5725ck7" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "name": "75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:26.413052673Z",
      "endTime": "2026-09-16T15:24:04.373688035Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv"
Unexpected error:
    <*fmt.wrapError | 0xc000d846a0>: 
    failed to create NodePool np-96ddmv: failed waiting for nodepool="np-96ddmv" for cluster "cluster-candidate-5-0-96ddmv" in resourcegroup="rg-candidate-5-0-dxgjw5725ck7" to finish creating: GET https://rp.j8680064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "name": "75dfa262-dc67-4818-8d1e-eacd838b47fd",
      "status": "Failed",
      "startTime": "2026-09-16T15:23:26.413052673Z",
      "endTime": "2026-09-16T15:24:04.373688035Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (5)
Date (UTC)Associated PRProw job
PR #69762100298527354654720
PR #69452100280971763912704
PR #69832100226936398680064
PR #69762100222840434331648
PR #65032100132038643290112
alert [svc] KubeDaemonSetRolloutStuck fired
full failure pattern:
alert [svc] KubeDaemonSetRolloutStuck fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert50.67%5 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 5none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T22:06:04Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.104:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1475712/providers/microsoft.monitor/accounts/services-j1475712", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-59mfc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T22:06:04Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.104:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1475712/providers/microsoft.monitor/accounts/services-j1475712", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-59mfc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:55:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7836672/providers/microsoft.monitor/accounts/services-j7836672", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-zdqrc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:55:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.73:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7836672/providers/microsoft.monitor/accounts/services-j7836672", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-5979b8454d-zdqrc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:45:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:45:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:46:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:46:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.65.38:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6091136/providers/microsoft.monitor/accounts/services-j6091136", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-574f657d96-qxmdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #68202100332359231475712
PR #68202100270038647836672
PR #69822100248871216091136
PR #68202100161854826352640
PR #68202100132066686406656
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert50.67%5 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 5none
Full failure examples (3)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-dgkdj", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-dgkdj has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-cb282", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-cb282 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-hdnbp", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-hdnbp has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:55:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1475712-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-f58cs", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-f58cs has been in a non-ready state for longer than 5 minutes.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-59s8j", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-59s8j has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-9zlkb", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-9zlkb has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-t8hdl", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-t8hdl has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-16T17:45:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7836672-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-49tpv", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-49tpv has been in a non-ready state for longer than 5 minutes.
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:36:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-cwxxr", region="canadacentral", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-cwxxr has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-16T16:36:14Z
  Ended:       2026-09-16T17:19:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-28181016-vmss000000-vxff5", region="canadacentral", severity="warning"
  Description: Pod kube-system/eraser-aks-system-28181016-vmss000000-vxff5 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-16T16:36:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6091136-mgmt-1", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-xfmjp", region="canadacentral", severity="warning"
  Description: Pod kube-system/ama-metrics-node-xfmjp has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #68202100332359231475712
PR #68202100270038647836672
PR #69822100248871216091136
PR #68202100161854826352640
PR #68202100132066686406656
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; host...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e40.53%4 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 6none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:90]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000eaac80>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-5sd5p2zrd22p" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb",
      "name": "cb287309-affe-4cb6-ba60-0ab97101fffb",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:15.559693502Z",
      "endTime": "2026-09-16T21:51:23.369287505Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:90]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000eaac80>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-5sd5p2zrd22p" to finish creating: GET https://rp.j3686400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cb287309-affe-4cb6-ba60-0ab97101fffb",
      "name": "cb287309-affe-4cb6-ba60-0ab97101fffb",
      "status": "Failed",
      "startTime": "2026-09-16T21:22:15.559693502Z",
      "endTime": "2026-09-16T21:51:23.369287505Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:584]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101ca40>: 
    failed to create HCP cluster np-downgrade-ptnt6v: failed waiting for cluster="np-downgrade-ptnt6v" in resourcegroup="rg-np-downgrade-ptnt6v-s6586bctkvvs" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "name": "b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:45.747660708Z",
      "endTime": "2026-09-16T15:36:50.286764481Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:584]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101ca40>: 
    failed to create HCP cluster np-downgrade-ptnt6v: failed waiting for cluster="np-downgrade-ptnt6v" in resourcegroup="rg-np-downgrade-ptnt6v-s6586bctkvvs" to finish creating: GET https://rp.j7769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "name": "b81923cc-2583-43fb-a0ea-baa197fb0d6f",
      "status": "Failed",
      "startTime": "2026-09-16T15:07:45.747660708Z",
      "endTime": "2026-09-16T15:36:50.286764481Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vm24lv8gf4qb/cluster-candidate-4-21-llvd77 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011b0380>: 
    failed to create HCP cluster cluster-candidate-4-21-llvd77: failed waiting for cluster="cluster-candidate-4-21-llvd77" in resourcegroup="rg-candidate-4-21-vm24lv8gf4qb" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "name": "a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.803955113Z",
      "endTime": "2026-09-16T11:21:09.695382854Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vm24lv8gf4qb/cluster-candidate-4-21-llvd77 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011b0380>: 
    failed to create HCP cluster cluster-candidate-4-21-llvd77: failed waiting for cluster="cluster-candidate-4-21-llvd77" in resourcegroup="rg-candidate-4-21-vm24lv8gf4qb" to finish creating: GET https://rp.j5236352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "name": "a4429abc-10d8-4151-ad1a-6f415aa5475f",
      "status": "Failed",
      "startTime": "2026-09-16T10:52:02.803955113Z",
      "endTime": "2026-09-16T11:21:09.695382854Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #63642100324480323686400
PR #63642100229912697769984
PR #67912100161923835236352
PR #68792100132080049459200
alert [svc] ServiceMemoryDrift fired
full failure pattern:
alert [svc] ServiceMemoryDrift fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert40.53%4 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 4none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:38:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j6757888-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-bcnth", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-bcnth (namespace arobit) on cluster ci01-j6757888-svc has averaged 127.2% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-16T21:58:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9922816-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8kkxq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8kkxq (namespace arobit) on cluster ci01-j9922816-svc has averaged 125.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-16T19:40:11Z
  Ended:       2026-09-16T19:46:13Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j2121088-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rxj44", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rxj44 (namespace arobit) on cluster ci01-j2121088-svc has averaged 126.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #69452100313784206757888
PR #68942100309676229922816
PR #63642100285588732121088
PR #68942100273649465430016
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e40.53%4 of 751 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #69832100226936398680064
PR #67912100196783199096832
PR #67912100161923835236352
PR #68792100132080049459200

Environment: INT

Failure patterns: 1 · Failures matched: 1 · Total runs: 27 · Runs affected: 1

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.70%1 of 27 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1DEV, PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000df6cd8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-pfttlc9b26s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "name": "ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "status": "Failed",
      "startTime": "2026-09-16T22:57:32.075098968Z",
      "endTime": "2026-09-16T23:16:32.974740274Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000df6cd8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-pfttlc9b26s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "name": "ff088bfb-6289-4eb8-a3c7-2bd5dc2e8821",
      "status": "Failed",
      "startTime": "2026-09-16T22:57:32.075098968Z",
      "endTime": "2026-09-16T23:16:32.974740274Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100351631508901888

Environment: PROD

Failure patterns: 5 · Failures matched: 9 · Total runs: 22 · Runs affected: 2

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.55%1 of 22 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 5STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001d82db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-d2hkj9sg5shk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001d82db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-d2hkj9sg5shk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000fd73e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dqvpfm666hzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000fd73e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dqvpfm666hzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002c97050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8h5dwhv96zkm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002c97050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8h5dwhv96zkm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142039359488
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.55%1 of 22 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1DEV, INT, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f3cd80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b7c4p2nvk6d6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d",
      "name": "0259344c-f395-4714-abc8-4abafaaf707d",
      "status": "Failed",
      "startTime": "2026-09-16T23:18:34.031156052Z",
      "endTime": "2026-09-16T23:37:37.814154784Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f3cd80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-b7c4p2nvk6d6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0259344c-f395-4714-abc8-4abafaaf707d",
      "name": "0259344c-f395-4714-abc8-4abafaaf707d",
      "status": "Failed",
      "startTime": "2026-09-16T23:18:34.031156052Z",
      "endTime": "2026-09-16T23:37:37.814154784Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142039359488
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.55%1 of 22 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d85560>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jzkmnxzs9c52": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jzkmnxzs9c52/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d85560>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jzkmnxzs9c52": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jzkmnxzs9c52/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142039359488
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.55%1 of 22 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00152ab10>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00152ab10>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142039359488
cluster operators not available: console (no Available condition), dns (Available=False, Progressing=True, Degraded=True, Upgradeable=True), image-registry (Available=False, Progr...
full failure pattern:
cluster operators not available: console (no Available condition), dns (Available=False, Progressing=True, Degraded=True, Upgradeable=True), image-registry (Available=False, Progressing=True, Degraded=True), ingress (Available=False, Progressing=True
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.55%1 of 22 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002a4e018>: 
    context deadline exceeded
    cluster operators not available: console (no Available condition), dns (Available=False, Progressing=True, Degraded=True, Upgradeable=True), image-registry (Available=False, Progressing=True, Degraded=True), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (no Available condition), openshift-samples (no Available condition), storage (Degraded=False, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002a4e018>: 
    context deadline exceeded
    cluster operators not available: console (no Available condition), dns (Available=False, Progressing=True, Degraded=True, Upgradeable=True), image-registry (Available=False, Progressing=True, Degraded=True), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False), monitoring (no Available condition), openshift-samples (no Available condition), storage (Degraded=False, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown)
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100173746357669888

Environment: STG

Failure patterns: 5 · Failures matched: 12 · Total runs: 24 · Runs affected: 1

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.17%1 of 24 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 5PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc001d4bc80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-56f2xp2w67w8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc001d4bc80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-56f2xp2w67w8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0012fd950>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-57279mxs72dc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0012fd950>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-57279mxs72dc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000f2fd40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-g8sp2mm6ks67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000f2fd40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-g8sp2mm6ks67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142060331008
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.17%1 of 24 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc0004d4640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-c8fd7f" in resourcegroup="rg-np-version-upgrade-c8fd7f-kdnr5znbw57r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c8fd7f-kdnr5znbw57r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c8fd7f/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc0004d4640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-c8fd7f" in resourcegroup="rg-np-version-upgrade-c8fd7f-kdnr5znbw57r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c8fd7f-kdnr5znbw57r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c8fd7f/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl"
Unexpected error:
    <*fmt.wrapError | 0xc002c1a920>: 
    failed to create NodePool np-pbnkzl: failed starting nodepool creation "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl" in resourcegroup="rg-nightly-4-20-6c9ns9nbgrzm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-6c9ns9nbgrzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-pbnkzl/nodePools/np-pbnkzl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl"
Unexpected error:
    <*fmt.wrapError | 0xc002c1a920>: 
    failed to create NodePool np-pbnkzl: failed starting nodepool creation "np-pbnkzl" for cluster "cluster-nightly-4-20-pbnkzl" in resourcegroup="rg-nightly-4-20-6c9ns9nbgrzm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-6c9ns9nbgrzm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-pbnkzl/nodePools/np-pbnkzl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc000facdc0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-m58t8j" resourcegroup="rg-np-dg-minor-m58t8j-trxh29mwfdlb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-m58t8j-trxh29mwfdlb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-m58t8j/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-15-135250
Unexpected error:
    <*fmt.wrapError | 0xc000facdc0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-m58t8j" resourcegroup="rg-np-dg-minor-m58t8j-trxh29mwfdlb": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-m58t8j-trxh29mwfdlb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-m58t8j/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-15-135250\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142060331008
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.17%1 of 24 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1DEV, INT, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8"
Unexpected error:
    <*fmt.wrapError | 0xc00121ee40>: 
    failed to create NodePool np-krkxx8: failed waiting for nodepool="np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8" in resourcegroup="rg-nightly-4-23-7l9rdmzm7f2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "name": "4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "status": "Failed",
      "startTime": "2026-09-16T23:37:25.694607859Z",
      "endTime": "2026-09-16T23:56:27.737971571Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8"
Unexpected error:
    <*fmt.wrapError | 0xc00121ee40>: 
    failed to create NodePool np-krkxx8: failed waiting for nodepool="np-krkxx8" for cluster "cluster-nightly-4-23-krkxx8" in resourcegroup="rg-nightly-4-23-7l9rdmzm7f2z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "name": "4e59a6ba-7f62-48bf-8e29-7e6ddefd08f9",
      "status": "Failed",
      "startTime": "2026-09-16T23:37:25.694607859Z",
      "endTime": "2026-09-16T23:56:27.737971571Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142060331008
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.17%1 of 24 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010b5a80>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-2wnkdzqz6d9w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2wnkdzqz6d9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010b5a80>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-2wnkdzqz6d9w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2wnkdzqz6d9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142060331008
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.17%1 of 24 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 9: 0 · Sep 10: 0 · Sep 11: 0 · Sep 12: 0 · Sep 13: 0 · Sep 14: 0 · Sep 15: 0 · Sep 16: 1PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016e5a88>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016e5a88>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2100359142060331008