CIHealth
<
Weekly: Oct 12 - Oct 18▾
Last 7 Days Weekly: Oct 12 - Oct 18 Sprint: Oct 12 - Oct 25 Single Day: Oct 18
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 172 · Failures matched: 757 · Total runs: 491 · Runs affected: 340

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key "globalMSIId" not found
full failure pattern:
failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key "globalMSIId" not found
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision61.22%6 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 4 · Oct 17: 0 · Oct 18: 2none
Full failure examples (3)
time=2026-10-18T15:38:05.575Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-18T15:38:06.914Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found"
time=2026-10-18T12:10:56.819Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-18T12:10:57.618Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found"
time=2026-10-16T23:16:01.424Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-16T23:16:02.804Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #75802111837676591124480
PR #75802111776757118930944
PR #75802111233385828454400
PR #75802111219106693779456
PR #75802111146056094846976
PR #75802111136882900340736
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [infra] Cosmos DB Normalized RU Consumption Burst High - <cosmos-account> fired
full failure pattern:
alert [infra] Cosmos DB Normalized RU Consumption Burst High - <cosmos-account> fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert40.81%4 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 3 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T03:40:18Z
  Ended:       2026-10-14T03:43:19Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
Firing 2:
  State:       Resolved
  Started:     2026-10-14T03:52:13Z
  Ended:       2026-10-14T04:06:24Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T00:40:15Z
  Ended:       2026-10-14T01:10:12Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-13T22:03:21Z
  Ended:       2026-10-13T22:29:18Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
Contributing tests (4)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption Burst High - arohcpci01-rp-j0532352 does not fire1
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption Burst High - arohcpci01-rp-j2816128 does not fire1
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption Burst High - arohcpci01-rp-j6775168 does not fire1
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption Burst High - arohcpci01-rp-j7369856 does not fire1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #74952110199338117369856
PR #74952110153446450532352
PR #74952110115702512816128
PR #74952110080447856775168
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1
full failure pattern:
failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision40.81%4 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 2 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0none
Full failure examples (3)
time=2026-10-16T23:22:36.468Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-16T23:22:36.986Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:d3bdf5672c862ebc0a35ea497266f8c7598862b9377e076ed14ffc329be19981]"
time=2026-10-16T23:22:36.986Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-16T23:22:37.251Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-16T23:22:37.674Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-16T23:22:37.674Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-16T23:22:37.803Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-16T23:23:44.090Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-16T23:23:44.090Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-16T15:46:44.545Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-16T15:46:45.068Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:d3bdf5672c862ebc0a35ea497266f8c7598862b9377e076ed14ffc329be19981]"
time=2026-10-16T15:46:45.068Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-16T15:46:45.193Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-16T15:46:45.343Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-16T15:46:45.343Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-16T15:46:45.452Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-16T15:47:32.298Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-16T15:47:32.298Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-15T23:49:29.451Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n"
time=2026-10-15T23:49:29.912Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:d3bdf5672c862ebc0a35ea497266f8c7598862b9377e076ed14ffc329be19981]"
time=2026-10-15T23:49:29.912Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-15T23:49:30.191Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc
time=2026-10-15T23:49:30.484Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false
time=2026-10-15T23:49:30.485Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-15T23:49:30.579Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1
time=2026-10-15T23:50:28.126Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-15T23:50:28.126Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/storageclass (stamp=1)3
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/storageclass (stamp=2)1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #74512111228423002132480
PR #74512111110619204685824
PR #74512110868757877362688
PR #74512110747548531560448
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet po...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [dataPlaneOIDCFederation] data-plane OIDC federation not yet ensured for 3 operator(s): disk-csi-driver (/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver), file-csi-driver (/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver), image-registry (/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry); [hypershiftHostedCluster] hosted cluster state not cached yet; [roleAssignments] role assignments not yet confirmed; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 61 · Oct 17: 0 · Oct 18: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x3cd963448a00>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-g99wvvnb2gqt" to finish creating: GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/feb6dbe3-4518-4432-8ee1-e82bfe257e23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/feb6dbe3-4518-4432-8ee1-e82bfe257e23",
      "name": "feb6dbe3-4518-4432-8ee1-e82bfe257e23",
      "status": "Failed",
      "startTime": "2026-10-16T11:18:40.122769052Z",
      "endTime": "2026-10-16T11:47:49.816923189Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [dataPlaneOIDCFederation] data-plane OIDC federation not yet ensured for 3 operator(s): disk-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-51/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver), file-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-51/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver), image-registry (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-51/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry); [hypershiftHostedCluster] hosted cluster state not cached yet; [roleAssignments] role assignments not yet confirmed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/feb6dbe3-4518-4432-8ee1-e82bfe257e23
            correlation ID: 6e26fa23-6c8f-48c7-84e4-93c4480408ac
            client request ID: 0de8699c-e23a-4388-aef8-31bf31c0384d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x3cd963448a00>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-g99wvvnb2gqt" to finish creating: GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/feb6dbe3-4518-4432-8ee1-e82bfe257e23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/feb6dbe3-4518-4432-8ee1-e82bfe257e23",
      "name": "feb6dbe3-4518-4432-8ee1-e82bfe257e23",
      "status": "Failed",
      "startTime": "2026-10-16T11:18:40.122769052Z",
      "endTime": "2026-10-16T11:47:49.816923189Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [dataPlaneOIDCFederation] data-plane OIDC federation not yet ensured for 3 operator(s): disk-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-51/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver), file-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-51/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver), image-registry (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-51/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry); [hypershiftHostedCluster] hosted cluster state not cached yet; [roleAssignments] role assignments not yet confirmed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/feb6dbe3-4518-4432-8ee1-e82bfe257e23
            correlation ID: 6e26fa23-6c8f-48c7-84e4-93c4480408ac
            client request ID: 0de8699c-e23a-4388-aef8-31bf31c0384d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x390e05a68de0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-xqknpzfzmr5w" to finish creating: GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d23223-112b-4b7f-ba15-3c3397a86d65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d23223-112b-4b7f-ba15-3c3397a86d65",
      "name": "73d23223-112b-4b7f-ba15-3c3397a86d65",
      "status": "Failed",
      "startTime": "2026-10-16T11:18:15.514720241Z",
      "endTime": "2026-10-16T11:47:24.729519268Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [dataPlaneOIDCFederation] data-plane OIDC federation not yet ensured for 3 operator(s): disk-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-17/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver), file-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-17/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver), image-registry (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-17/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry); [hypershiftHostedCluster] hosted cluster state not cached yet; [roleAssignments] role assignments not yet confirmed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d23223-112b-4b7f-ba15-3c3397a86d65
            correlation ID: 88facd09-aef2-4a7f-9c7e-db87f515aac5
            client request ID: 4c75b22c-aded-487f-a952-f949adbe49c6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x390e05a68de0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-xqknpzfzmr5w" to finish creating: GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d23223-112b-4b7f-ba15-3c3397a86d65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d23223-112b-4b7f-ba15-3c3397a86d65",
      "name": "73d23223-112b-4b7f-ba15-3c3397a86d65",
      "status": "Failed",
      "startTime": "2026-10-16T11:18:15.514720241Z",
      "endTime": "2026-10-16T11:47:24.729519268Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [dataPlaneOIDCFederation] data-plane OIDC federation not yet ensured for 3 operator(s): disk-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-17/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver), file-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-17/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver), image-registry (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-17/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry); [hypershiftHostedCluster] hosted cluster state not cached yet; [roleAssignments] role assignments not yet confirmed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73d23223-112b-4b7f-ba15-3c3397a86d65
            correlation ID: 88facd09-aef2-4a7f-9c7e-db87f515aac5
            client request ID: 4c75b22c-aded-487f-a952-f949adbe49c6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-g5rdgl7zwjl6/cluster-candidate-4-20-k9b9wf should provision
Unexpected error:
    <*fmt.wrapError | 0x18f77dd205c0>: 
    failed to create HCP cluster cluster-candidate-4-20-k9b9wf: failed waiting for cluster="cluster-candidate-4-20-k9b9wf" in resourcegroup="rg-candidate-4-20-g5rdgl7zwjl6" to finish creating: GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0984890f-0a19-47d9-9a1c-275d44c6e314
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0984890f-0a19-47d9-9a1c-275d44c6e314",
      "name": "0984890f-0a19-47d9-9a1c-275d44c6e314",
      "status": "Failed",
      "startTime": "2026-10-16T11:18:36.959680763Z",
      "endTime": "2026-10-16T11:47:39.816204138Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [dataPlaneOIDCFederation] data-plane OIDC federation not yet ensured for 3 operator(s): disk-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-36/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver), file-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-36/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver), image-registry (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-36/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry); [hypershiftHostedCluster] hosted cluster state not cached yet; [roleAssignments] role assignments not yet confirmed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0984890f-0a19-47d9-9a1c-275d44c6e314
            correlation ID: c1d2f06b-ddfc-4fd2-8766-74fa80998c3b
            client request ID: ef5cac13-1a05-485d-b15a-d283225aac9d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-g5rdgl7zwjl6/cluster-candidate-4-20-k9b9wf should provision
Unexpected error:
    <*fmt.wrapError | 0x18f77dd205c0>: 
    failed to create HCP cluster cluster-candidate-4-20-k9b9wf: failed waiting for cluster="cluster-candidate-4-20-k9b9wf" in resourcegroup="rg-candidate-4-20-g5rdgl7zwjl6" to finish creating: GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0984890f-0a19-47d9-9a1c-275d44c6e314
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0984890f-0a19-47d9-9a1c-275d44c6e314",
      "name": "0984890f-0a19-47d9-9a1c-275d44c6e314",
      "status": "Failed",
      "startTime": "2026-10-16T11:18:36.959680763Z",
      "endTime": "2026-10-16T11:47:39.816204138Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [dataPlaneOIDCFederation] data-plane OIDC federation not yet ensured for 3 operator(s): disk-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-36/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver), file-csi-driver (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-36/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver), image-registry (/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-36/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry); [hypershiftHostedCluster] hosted cluster state not cached yet; [roleAssignments] role assignments not yet confirmed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9203200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0984890f-0a19-47d9-9a1c-275d44c6e314
            correlation ID: c1d2f06b-ddfc-4fd2-8766-74fa80998c3b
            client request ID: ef5cac13-1a05-485d-b15a-d283225aac9d
occurred
Contributing tests (61)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE starts a control plane z-stream upgrade with an SRE pin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69822111041799249203200
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(External): timeout must be > 0, got 0s
full failure pattern:
VerifyIngressControllerScope(External): timeout must be > 0, got 0s
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 4 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x26b7f5580860>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x26b7f5580860>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x5b6d94696d0>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x5b6d94696d0>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x362088f5cd60>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x362088f5cd60>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772110340691224694784
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRIngressController loadBalancer config is nil
full failure pattern:
IngressController loadBalancer config is nil
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 2 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x284568f46bc0>: 
    IngressController loadBalancer config is nil
    {
        s: "IngressController loadBalancer config is nil",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x284568f46bc0>: 
    IngressController loadBalancer config is nil
    {
        s: "IngressController loadBalancer config is nil",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x374a5e1794b0>: 
    IngressController loadBalancer config is nil
    {
        s: "IngressController loadBalancer config is nil",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x374a5e1794b0>: 
    IngressController loadBalancer config is nil
    {
        s: "IngressController loadBalancer config is nil",
    }
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772110129974156464128
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(External) timed out after 5m0
full failure pattern:
VerifyIngressControllerScope(External) timed out after 5m0
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 2 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x2c6abc8c6320>: 
    VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x2c6abc8c62a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x2c6abc8c6260>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x2c6abc8c6320>: 
    VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x2c6abc8c62a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x2c6abc8c6260>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x1ba452ab87c0>: 
    VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1ba452ab87a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1ba452ab8760>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x1ba452ab87c0>: 
    VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1ba452ab87a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1ba452ab8760>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772110235250528358400
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail code KeyVaultAccessForbidden; detail message Unable to access key vault resource '<url>' to enable encryption at rest. Please grant get, wrap ...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail code KeyVaultAccessForbidden; detail message Unable to access key vault resource '<url>' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x14dd1e091060>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-kdz84x79jt45" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "name": "75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "status": "Failed",
      "startTime": "2026-10-12T12:59:06.888844222Z",
      "endTime": "2026-10-12T13:28:08.456404461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine des-encrypt-des-np-f8895-79vxf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine des-encrypt-des-np-f8895-gfvrv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine des-encrypt-des-np-f8895-79vxf: Failed: virtualmachine failed to create or update. err: failed to create or update resource des-encrypt-kdz84x79jt45--managed/des-encrypt-des-np-f8895-79vxf (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-kdz84x79jt45--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-f8895-79vxf\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: KeyVaultAccessForbidden\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"KeyVaultAccessForbidden\",\n    \"message\": \"Unable to access key vault resource 'https://cust-kv-knxkedtao2ry2.vault.azure.net/keys/des-encryption-key/3126b361c5834ce5ad1e97723eb63855' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
            correlation ID: bd8a92be-4962-42e7-8e34-639e73d20045
            client request ID: 6eda585f-7541-486b-8dad-d5cd134191ff
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:151]: failed to create nodepool des-np with DES
Unexpected error:
    <*fmt.wrapError | 0x14dd1e091060>: 
    failed to create NodePool des-np: failed waiting for nodepool="des-np" for cluster "des-encrypt" in resourcegroup="des-encrypt-kdz84x79jt45" to finish creating: GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "name": "75f3427d-1a2b-4425-b9b0-7480d4e0b89b",
      "status": "Failed",
      "startTime": "2026-10-12T12:59:06.888844222Z",
      "endTime": "2026-10-12T13:28:08.456404461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine des-encrypt-des-np-f8895-79vxf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine des-encrypt-des-np-f8895-gfvrv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine des-encrypt-des-np-f8895-79vxf: Failed: virtualmachine failed to create or update. err: failed to create or update resource des-encrypt-kdz84x79jt45--managed/des-encrypt-des-np-f8895-79vxf (service: virtualmachine): PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-kdz84x79jt45--managed/providers/Microsoft.Compute/virtualMachines/des-encrypt-des-np-f8895-79vxf\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: KeyVaultAccessForbidden\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"KeyVaultAccessForbidden\",\n    \"message\": \"Unable to access key vault resource 'https://cust-kv-knxkedtao2ry2.vault.azure.net/keys/des-encryption-key/3126b361c5834ce5ad1e97723eb63855' to enable encryption at rest. Please grant get, wrap and unwrap key permissions to disk encryption set 'des... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5598976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75f3427d-1a2b-4425-b9b0-7480d4e0b89b
            correlation ID: bd8a92be-4962-42e7-8e34-639e73d20045
            client request ID: 6eda585f-7541-486b-8dad-d5cd134191ff
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74702109611047785598976
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after a... message truncated; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x2dbe92ada858>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-t6jslt7lwjkt" to finish creating: GET https://rp.j2382080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/88e58151-a475-4725-8e82-2d51b0f7fb69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/88e58151-a475-4725-8e82-2d51b0f7fb69",
      "name": "88e58151-a475-4725-8e82-2d51b0f7fb69",
      "status": "Failed",
      "startTime": "2026-10-16T13:50:29.440992073Z",
      "endTime": "2026-10-16T14:09:32.07987309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-5p4h6-brpt4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-t6jslt7lwjkt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-5p4h6-brpt4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-5p4h6-brpt4: Failed: availabilitysets failed to create or update. err: failed to get existing resource cni-cilium-t6jslt7lwjkt--managed/2tftmlmmdno0igptnhfrk6md9uivpr81_cilium-cl-cilium-np-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-t6jslt7lwjkt--managed/providers/Microsoft.Compute/availabilitySets/2tftmlmmdno0igptnhfrk6md9uivpr81_cilium-cl-cilium-np-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '10' seconds after a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2382080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/88e58151-a475-4725-8e82-2d51b0f7fb69
            correlation ID: 054f0541-4905-4acf-9baf-353f2e747fdd
            client request ID: 320b7448-1575-4fdb-9611-9bf748713191
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x2dbe92ada858>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-t6jslt7lwjkt" to finish creating: GET https://rp.j2382080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/88e58151-a475-4725-8e82-2d51b0f7fb69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/88e58151-a475-4725-8e82-2d51b0f7fb69",
      "name": "88e58151-a475-4725-8e82-2d51b0f7fb69",
      "status": "Failed",
      "startTime": "2026-10-16T13:50:29.440992073Z",
      "endTime": "2026-10-16T14:09:32.07987309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-5p4h6-brpt4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-t6jslt7lwjkt--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-5p4h6-brpt4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-5p4h6-brpt4: Failed: availabilitysets failed to create or update. err: failed to get existing resource cni-cilium-t6jslt7lwjkt--managed/2tftmlmmdno0igptnhfrk6md9uivpr81_cilium-cl-cilium-np-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-t6jslt7lwjkt--managed/providers/Microsoft.Compute/availabilitySets/2tftmlmmdno0igptnhfrk6md9uivpr81_cilium-cl-cilium-np-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '10' seconds after a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2382080.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/88e58151-a475-4725-8e82-2d51b0f7fb69
            correlation ID: 054f0541-4905-4acf-9baf-353f2e747fdd
            client request ID: 320b7448-1575-4fdb-9611-9bf748713191
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75562111079254782382080
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available. ... message truncated; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0x2fa3cd2a3c20>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-wl8xt5tlkcq7" to finish creating: GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/63825d69-d610-4421-b3ee-50ac654908c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/63825d69-d610-4421-b3ee-50ac654908c5",
      "name": "63825d69-d610-4421-b3ee-50ac654908c5",
      "status": "Failed",
      "startTime": "2026-10-16T21:00:14.715888281Z",
      "endTime": "2026-10-16T21:19:17.454369198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-rnlcf-ttbgr: Failed: interfaces failed to create or update. err: failed to get existing resource external-auth-cluster-wl8xt5tlkcq7--managed/ea-cluster-ea-np-1-rnlcf-ttbgr-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wl8xt5tlkcq7--managed/providers/Microsoft.Network/networkInterfaces/ea-cluster-ea-np-1-rnlcf-ttbgr-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/63825d69-d610-4421-b3ee-50ac654908c5
            correlation ID: 2667b7c1-2456-4fb3-a1b3-9f090199ed55
            client request ID: 1483f922-ab7c-4902-933a-1a3aef1cf957
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0x2fa3cd2a3c20>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-wl8xt5tlkcq7" to finish creating: GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/63825d69-d610-4421-b3ee-50ac654908c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/63825d69-d610-4421-b3ee-50ac654908c5",
      "name": "63825d69-d610-4421-b3ee-50ac654908c5",
      "status": "Failed",
      "startTime": "2026-10-16T21:00:14.715888281Z",
      "endTime": "2026-10-16T21:19:17.454369198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-rnlcf-ttbgr: Failed: interfaces failed to create or update. err: failed to get existing resource external-auth-cluster-wl8xt5tlkcq7--managed/ea-cluster-ea-np-1-rnlcf-ttbgr-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wl8xt5tlkcq7--managed/providers/Microsoft.Network/networkInterfaces/ea-cluster-ea-np-1-rnlcf-ttbgr-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/63825d69-d610-4421-b3ee-50ac654908c5
            correlation ID: 2667b7c1-2456-4fb3-a1b3-9f090199ed55
            client request ID: 1483f922-ab7c-4902-933a-1a3aef1cf957
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75312111186341810147328
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available. Refer ... message truncated Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x33c730fe0780>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2q8nt57gw7c4" to finish creating: GET https://rp.j3035776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f78025-9771-4e07-9a76-cc316ad5bcaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f78025-9771-4e07-9a76-cc316ad5bcaa",
      "name": "f4f78025-9771-4e07-9a76-cc316ad5bcaa",
      "status": "Failed",
      "startTime": "2026-10-16T23:21:44.60401103Z",
      "endTime": "2026-10-16T23:40:48.245826795Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9v1y5n5j8r9i0x-np-1-89jk5-zffrv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2q8nt57gw7c4--managed/providers/Microsoft.Compute/virtualMachines/p9v1y5n5j8r9i0x-np-1-89jk5-zffrv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9v1y5n5j8r9i0x-np-1-89jk5-vpf5w: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-2q8nt57gw7c4--managed/p9v1y5n5j8r9i0x-np-1-89jk5-vpf5w-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2q8nt57gw7c4--managed/providers/Microsoft.Network/networkInterfaces/p9v1y5n5j8r9i0x-np-1-89jk5-vpf5w-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3d8c36e1-ae7a-42ef-b7d1-ea4667708d30' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\nMachine p9v1y5n5j8r9i0x-np-1-89jk5-zffrv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2q8nt57gw7c4--managed/providers/Microsoft.Compute/virtualMachines/p9v1y5n5j8r9i0x-np-1-89jk5-zffrv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3035776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f78025-9771-4e07-9a76-cc316ad5bcaa
            correlation ID: 57a29177-39d2-4d3f-ac56-44b9805e57f9
            client request ID: 8065bb9a-6902-4747-a2b4-2914d4dc8e70
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x33c730fe0780>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2q8nt57gw7c4" to finish creating: GET https://rp.j3035776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f78025-9771-4e07-9a76-cc316ad5bcaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f78025-9771-4e07-9a76-cc316ad5bcaa",
      "name": "f4f78025-9771-4e07-9a76-cc316ad5bcaa",
      "status": "Failed",
      "startTime": "2026-10-16T23:21:44.60401103Z",
      "endTime": "2026-10-16T23:40:48.245826795Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p9v1y5n5j8r9i0x-np-1-89jk5-zffrv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2q8nt57gw7c4--managed/providers/Microsoft.Compute/virtualMachines/p9v1y5n5j8r9i0x-np-1-89jk5-zffrv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p9v1y5n5j8r9i0x-np-1-89jk5-vpf5w: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-2q8nt57gw7c4--managed/p9v1y5n5j8r9i0x-np-1-89jk5-vpf5w-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2q8nt57gw7c4--managed/providers/Microsoft.Network/networkInterfaces/p9v1y5n5j8r9i0x-np-1-89jk5-vpf5w-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3d8c36e1-ae7a-42ef-b7d1-ea4667708d30' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\nMachine p9v1y5n5j8r9i0x-np-1-89jk5-zffrv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2q8nt57gw7c4--managed/providers/Microsoft.Compute/virtualMachines/p9v1y5n5j8r9i0x-np-1-89jk5-zffrv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3035776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4f78025-9771-4e07-9a76-cc316ad5bcaa
            correlation ID: 57a29177-39d2-4d3f-ac56-44b9805e57f9
            client request ID: 8065bb9a-6902-4747-a2b4-2914d4dc8e70
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75312111218975433035776
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available. Refer ... message truncated; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x33cb56e17020>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4xcd2z54mnjl" to finish creating: GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed1bfb49-b931-4647-a8ac-58b583c3b50b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed1bfb49-b931-4647-a8ac-58b583c3b50b",
      "name": "ed1bfb49-b931-4647-a8ac-58b583c3b50b",
      "status": "Failed",
      "startTime": "2026-10-16T10:16:52.135966436Z",
      "endTime": "2026-10-16T10:35:53.058795763Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s2n7n1o2s1t5d5z-np-1-m5qjf-pw27v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s2n7n1o2s1t5d5z-np-1-m5qjf-49zn4: Failed: virtualmachine failed to create or update. err: failed to get existing resource private-keyvault-4xcd2z54mnjl--managed/s2n7n1o2s1t5d5z-np-1-m5qjf-49zn4 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xcd2z54mnjl--managed/providers/Microsoft.Compute/virtualMachines/s2n7n1o2s1t5d5z-np-1-m5qjf-49zn4\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3871b527-fb1e-4123-b38b-3cb2445a9fc8' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed1bfb49-b931-4647-a8ac-58b583c3b50b
            correlation ID: ef5c7325-64f6-4ca1-87ca-c2c3e96bf69a
            client request ID: 392329dd-b2a3-4225-a4e8-154f2f223edb
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x33cb56e17020>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4xcd2z54mnjl" to finish creating: GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed1bfb49-b931-4647-a8ac-58b583c3b50b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed1bfb49-b931-4647-a8ac-58b583c3b50b",
      "name": "ed1bfb49-b931-4647-a8ac-58b583c3b50b",
      "status": "Failed",
      "startTime": "2026-10-16T10:16:52.135966436Z",
      "endTime": "2026-10-16T10:35:53.058795763Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s2n7n1o2s1t5d5z-np-1-m5qjf-pw27v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s2n7n1o2s1t5d5z-np-1-m5qjf-49zn4: Failed: virtualmachine failed to create or update. err: failed to get existing resource private-keyvault-4xcd2z54mnjl--managed/s2n7n1o2s1t5d5z-np-1-m5qjf-49zn4 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xcd2z54mnjl--managed/providers/Microsoft.Compute/virtualMachines/s2n7n1o2s1t5d5z-np-1-m5qjf-49zn4\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3871b527-fb1e-4123-b38b-3cb2445a9fc8' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ed1bfb49-b931-4647-a8ac-58b583c3b50b
            correlation ID: ef5c7325-64f6-4ca1-87ca-c2c3e96bf69a
            client request ID: 392329dd-b2a3-4225-a4e8-154f2f223edb
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75742111024523175268352
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, olm-operator, packageserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mvkgs4" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2777a0f273c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mvkgs4: failed waiting for cluster="cp-ystream-upgrade-4-22-mvkgs4" in resourcegroup="rg-cp-ystream-upgrade-4-22-blssk6qzfvws" to finish creating: GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "name": "a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:53.687204284Z",
      "endTime": "2026-10-12T16:21:54.576185121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-policy-controller, kube-controller-manager, openshift-apiserver, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, certified-operators-catalog, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
            correlation ID: f4045377-6462-4996-b798-a9fd2a096fae
            client request ID: 01fbf192-7565-417e-a2b4-3d30deb4b951
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mvkgs4" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x2777a0f273c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mvkgs4: failed waiting for cluster="cp-ystream-upgrade-4-22-mvkgs4" in resourcegroup="rg-cp-ystream-upgrade-4-22-blssk6qzfvws" to finish creating: GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "name": "a49f2790-97bb-4d56-b4a6-e43cb33e230e",
      "status": "Failed",
      "startTime": "2026-10-12T15:52:53.687204284Z",
      "endTime": "2026-10-12T16:21:54.576185121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, cluster-policy-controller, kube-controller-manager, openshift-apiserver, cluster-image-registry-operator, redhat-operators-catalog, community-operators-catalog, openshift-controller-manager, konnectivity-agent, olm-operator, packageserver, certified-operators-catalog, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, cluster-node-tuning-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4380032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a49f2790-97bb-4d56-b4a6-e43cb33e230e
            correlation ID: f4045377-6462-4996-b798-a9fd2a096fae
            client request ID: 01fbf192-7565-417e-a2b4-3d30deb4b951
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73832109662114334380032
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingOnInfrastructureReady: Cluster infrastructure is still provisioning; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-w5zxw6n76pst/cluster-candidate-4-20-dnk6t4 should provision
Unexpected error:
    <*fmt.wrapError | 0x4d01c384420>: 
    failed to create HCP cluster cluster-candidate-4-20-dnk6t4: failed waiting for cluster="cluster-candidate-4-20-dnk6t4" in resourcegroup="rg-candidate-4-20-w5zxw6n76pst" to finish creating: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
      "name": "d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
      "status": "Failed",
      "startTime": "2026-10-13T12:39:39.471658263Z",
      "endTime": "2026-10-13T13:08:48.195755915Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingOnInfrastructureReady: Cluster infrastructure is still provisioning; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
            correlation ID: 6fd24ae1-3485-4cf6-8f00-56a00457a1d3
            client request ID: 52d0a7ef-5194-4884-9aba-b6475a9fa220
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-w5zxw6n76pst/cluster-candidate-4-20-dnk6t4 should provision
Unexpected error:
    <*fmt.wrapError | 0x4d01c384420>: 
    failed to create HCP cluster cluster-candidate-4-20-dnk6t4: failed waiting for cluster="cluster-candidate-4-20-dnk6t4" in resourcegroup="rg-candidate-4-20-w5zxw6n76pst" to finish creating: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
      "name": "d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
      "status": "Failed",
      "startTime": "2026-10-13T12:39:39.471658263Z",
      "endTime": "2026-10-13T13:08:48.195755915Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingOnInfrastructureReady: Cluster infrastructure is still provisioning; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
            correlation ID: 6fd24ae1-3485-4cf6-8f00-56a00457a1d3
            client request ID: 52d0a7ef-5194-4884-9aba-b6475a9fa220
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74852109972734464233472
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Un...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x495f5a50100>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-vtqpftbtl8pd" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "name": "b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.565546726Z",
      "endTime": "2026-10-12T16:34:19.299056423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
            correlation ID: 2f0e8af3-4477-4dd1-b4a2-0bb21a3ecc71
            client request ID: af43345c-178c-4f70-985a-42472759437b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x495f5a50100>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-vtqpftbtl8pd" to finish creating: GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "name": "b568c3f0-0cc4-47ab-8959-64f2a472fc67",
      "status": "Failed",
      "startTime": "2026-10-12T16:05:12.565546726Z",
      "endTime": "2026-10-12T16:34:19.299056423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterValidation] DataPlaneIdentitiesPermissionsValidation: InternalError: Unable to verify data plane identities permissions.; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9626368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b568c3f0-0cc4-47ab-8959-64f2a472fc67
            correlation ID: 2f0e8af3-4477-4dd1-b4a2-0bb21a3ecc71
            client request ID: af43345c-178c-4f70-985a-42472759437b
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772109664686109626368
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0xe13d9ba2880>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-9czhr6hdgkds" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "name": "abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:22.752877991Z",
      "endTime": "2026-10-12T19:14:30.175613398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
            correlation ID: e99f3924-5824-4d14-bfb3-a5c2c635c6fb
            client request ID: 3e3fb9a9-7a4c-4e3b-9ae6-64fae2a5931a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:135]: failed to create HCP cluster des-encrypt
Unexpected error:
    <*fmt.wrapError | 0xe13d9ba2880>: 
    failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-9czhr6hdgkds" to finish creating: GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "name": "abdd1b6c-8284-4503-9367-9545ab4b7e3c",
      "status": "Failed",
      "startTime": "2026-10-12T18:45:22.752877991Z",
      "endTime": "2026-10-12T19:14:30.175613398Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 20m44s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6169984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/abdd1b6c-8284-4503-9367-9545ab4b7e3c
            correlation ID: e99f3924-5824-4d14-bfb3-a5c2c635c6fb
            client request ID: 3e3fb9a9-7a4c-4e3b-9ae6-64fae2a5931a
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74672109704412896169984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2e793540e048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4b88pr2h6f62" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe",
      "name": "02548047-b3fd-49b8-8426-7df711bcbfbe",
      "status": "Failed",
      "startTime": "2026-10-14T10:01:14.415831901Z",
      "endTime": "2026-10-14T10:30:17.083278262Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-794ws-42z9w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-794ws-42z9w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
            correlation ID: d40a80e8-ffc2-4aca-bd50-98d7806c38e8
            client request ID: 84b1cf39-1ee3-4c6a-80ac-30f66febe153
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2e793540e048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4b88pr2h6f62" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe",
      "name": "02548047-b3fd-49b8-8426-7df711bcbfbe",
      "status": "Failed",
      "startTime": "2026-10-14T10:01:14.415831901Z",
      "endTime": "2026-10-14T10:30:17.083278262Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-794ws-42z9w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-794ws-42z9w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
            correlation ID: d40a80e8-ffc2-4aca-bd50-98d7806c38e8
            client request ID: 84b1cf39-1ee3-4c6a-80ac-30f66febe153
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75002110291728471691264
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-<id> to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x346ef5b24080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xn2t4zqk2qxz" to finish creating: GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "name": "439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "status": "Failed",
      "startTime": "2026-10-14T12:55:20.912229682Z",
      "endTime": "2026-10-14T13:24:20.249835656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-rpntb-stl8c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-rpntb-stl8c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
            correlation ID: eefc14a4-21ec-420e-9b63-e8df580da42a
            client request ID: b9d8fbaf-fe74-4488-8491-9a8ef5ba8fd7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x346ef5b24080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xn2t4zqk2qxz" to finish creating: GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "name": "439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "status": "Failed",
      "startTime": "2026-10-14T12:55:20.912229682Z",
      "endTime": "2026-10-14T13:24:20.249835656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-rpntb-stl8c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-rpntb-stl8c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
            correlation ID: eefc14a4-21ec-420e-9b63-e8df580da42a
            client request ID: b9d8fbaf-fe74-4488-8491-9a8ef5ba8fd7
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75252110338041116626944
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xcf2616b6280>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mfxbjq8jns8s" to finish creating: GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "name": "6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "status": "Failed",
      "startTime": "2026-10-14T07:49:18.1783953Z",
      "endTime": "2026-10-14T08:18:20.880236175Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-cw6kh-fqbnw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-cw6kh-fqbnw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
            correlation ID: 39af48df-821f-41b4-9eda-383af9fbc767
            client request ID: d214e485-a699-4f3c-bb9e-2d4e1497eed9
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xcf2616b6280>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mfxbjq8jns8s" to finish creating: GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "name": "6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "status": "Failed",
      "startTime": "2026-10-14T07:49:18.1783953Z",
      "endTime": "2026-10-14T08:18:20.880236175Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-cw6kh-fqbnw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-cw6kh-fqbnw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
            correlation ID: 39af48df-821f-41b4-9eda-383af9fbc767
            client request ID: d214e485-a699-4f3c-bb9e-2d4e1497eed9
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67122110261282979450880
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x1d04d19fc660>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-kjxkhz" in resourcegroup="rg-np-version-upgrade-kjxkhz-ldkgrb6cvc64" to finish creating: GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eec0983a-6d6f-417c-a350-df06c778abae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eec0983a-6d6f-417c-a350-df06c778abae",
      "name": "eec0983a-6d6f-417c-a350-df06c778abae",
      "status": "Failed",
      "startTime": "2026-10-16T21:16:56.042187584Z",
      "endTime": "2026-10-16T21:35:57.540237831Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o9t1p7r8v5x0r1j-npupgrade-4-20-jqtb8-n8bff: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o9t1p7r8v5x0r1j-npupgrade-4-20-jqtb8-n8bff: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eec0983a-6d6f-417c-a350-df06c778abae
            correlation ID: 62969ebc-fdd1-4a1a-808f-bd02fcd158bd
            client request ID: f0c87d09-298f-426d-885a-00fac76d2bf1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
    <*fmt.wrapError | 0x1d04d19fc660>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-kjxkhz" in resourcegroup="rg-np-version-upgrade-kjxkhz-ldkgrb6cvc64" to finish creating: GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eec0983a-6d6f-417c-a350-df06c778abae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eec0983a-6d6f-417c-a350-df06c778abae",
      "name": "eec0983a-6d6f-417c-a350-df06c778abae",
      "status": "Failed",
      "startTime": "2026-10-16T21:16:56.042187584Z",
      "endTime": "2026-10-16T21:35:57.540237831Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine o9t1p7r8v5x0r1j-npupgrade-4-20-jqtb8-n8bff: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine o9t1p7r8v5x0r1j-npupgrade-4-20-jqtb8-n8bff: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0147328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eec0983a-6d6f-417c-a350-df06c778abae
            correlation ID: 62969ebc-fdd1-4a1a-808f-bd02fcd158bd
            client request ID: f0c87d09-298f-426d-885a-00fac76d2bf1
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75312111186341810147328
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1eddd58959c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-brlqmmsvjbp7" to finish creating: GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ec84baa-1f08-44e4-8884-c7d91686d7f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ec84baa-1f08-44e4-8884-c7d91686d7f5",
      "name": "1ec84baa-1f08-44e4-8884-c7d91686d7f5",
      "status": "Failed",
      "startTime": "2026-10-16T10:26:32.519187505Z",
      "endTime": "2026-10-16T10:45:34.937263417Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-brlqmmsvjbp7--managed/providers/Microsoft.Compute/virtualMachines/f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-brlqmmsvjbp7--managed/providers/Microsoft.Compute/virtualMachines/f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ec84baa-1f08-44e4-8884-c7d91686d7f5
            correlation ID: 6fd1b613-3112-4dbd-9dff-6f44c7554f29
            client request ID: 0ac1b9cb-8173-4c03-9f5c-ab7b1db7762b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1eddd58959c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-brlqmmsvjbp7" to finish creating: GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ec84baa-1f08-44e4-8884-c7d91686d7f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ec84baa-1f08-44e4-8884-c7d91686d7f5",
      "name": "1ec84baa-1f08-44e4-8884-c7d91686d7f5",
      "status": "Failed",
      "startTime": "2026-10-16T10:26:32.519187505Z",
      "endTime": "2026-10-16T10:45:34.937263417Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-brlqmmsvjbp7--managed/providers/Microsoft.Compute/virtualMachines/f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-brlqmmsvjbp7--managed/providers/Microsoft.Compute/virtualMachines/f1s8b5f8y5z5b5s-autoscale-noaz-4zlkx-pm4sh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5268352.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1ec84baa-1f08-44e4-8884-c7d91686d7f5
            correlation ID: 6fd1b613-3112-4dbd-9dff-6f44c7554f29
            client request ID: 0ac1b9cb-8173-4c03-9f5c-ab7b1db7762b
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75742111024523175268352
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code DeploymentActive; detail message The deployment with resource id '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '<timestamp>' with correlationId '<uuid>', and will expire at '<timestamp>' if it does not complete before then.
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:88]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xd4464611460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-b4xvfj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"DeploymentActive\",\r\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2hklr4tmfmv6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '10/12/2026 6:44:17 PM' with correlationId '483786d1-9a79-473c-9660-1216b08e7f90', and will expire at '10/19/2026 6:44:17 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
            correlation ID: d9670e5b-9049-4df2-b486-a7bed21f23b8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:88]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xd4464611460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-b4xvfj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"DeploymentActive\",\r\n    \"message\": \"The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-2hklr4tmfmv6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '10/12/2026 6:44:17 PM' with correlationId '483786d1-9a79-473c-9660-1216b08e7f90', and will expire at '10/19/2026 6:44:17 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-b4xvfj/operationStatuses/08584097762288469100
            correlation ID: d9670e5b-9049-4df2-b486-a7bed21f23b8
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74672109704412896169984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroup...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Authorization/roleAssignments/<uuid>' at line '<line>' and column '<column>'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'; provider Microsoft.Authorization
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create customer resources for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x3398cf888a80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-qzrlv9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Authorization/roleAssignments/adc91f81-bb61-5a2b-b4cc-0e8c1c460b92' at line '1' and column '27135'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 27135,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
            correlation ID: 9860797c-226a-442a-b3bd-c7b7c62b766b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create customer resources for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x3398cf888a80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-qzrlv9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Authorization/roleAssignments/adc91f81-bb61-5a2b-b4cc-0e8c1c460b92' at line '1' and column '27135'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 27135,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
            correlation ID: 9860797c-226a-442a-b3bd-c7b7c62b766b
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69202109978799176159232
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x12402d3b2b10>: 
    resource group "rg-candidate-4-23-qt7g4z697d5b": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-ch4n9f" in resourcegroup="rg-candidate-4-23-qt7g4z697d5b" to finish deleting: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1",
      "name": "27faaa2f-8540-458b-a125-3f3536544cd1",
      "status": "Failed",
      "startTime": "2026-10-13T12:59:59.269440518Z",
      "endTime": "2026-10-13T13:24:08.339110012Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdtgm7u9ipde1epc2rt5pe43isp14m7 still exists (deletion dispatched at 2026-10-13T12:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
            correlation ID: 365de814-fd9f-4b17-8525-a2096af1cada
            client request ID: 166f0483-5054-4255-b484-02e488295af5
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
    <*errors.joinError | 0x12402d3b2b10>: 
    resource group "rg-candidate-4-23-qt7g4z697d5b": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-ch4n9f" in resourcegroup="rg-candidate-4-23-qt7g4z697d5b" to finish deleting: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1",
      "name": "27faaa2f-8540-458b-a125-3f3536544cd1",
      "status": "Failed",
      "startTime": "2026-10-13T12:59:59.269440518Z",
      "endTime": "2026-10-13T13:24:08.339110012Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdtgm7u9ipde1epc2rt5pe43isp14m7 still exists (deletion dispatched at 2026-10-13T12:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
            correlation ID: 365de814-fd9f-4b17-8525-a2096af1cada
            client request ID: 166f0483-5054-4255-b484-02e488295af5
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74852109972734464233472
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRGet "<url>": context deadline exceeded
full failure pattern:
Get "<url>": context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.0ju5.j8150016.hcp.osadev.cloud": context deadline exceeded
Unexpected error:
    <*url.Error | 0x36a4b7cb55f0>: 
    Get "https://console-openshift-console.apps.aro.private-kas.0ju5.j8150016.hcp.osadev.cloud": context deadline exceeded
    {
        Op: "Get",
        URL: "https://console-openshift-console.apps.aro.private-kas.0ju5.j8150016.hcp.osadev.cloud",
        Err: <context.deadlineExceededError>{},
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.0ju5.j8150016.hcp.osadev.cloud": context deadline exceeded
Unexpected error:
    <*url.Error | 0x36a4b7cb55f0>: 
    Get "https://console-openshift-console.apps.aro.private-kas.0ju5.j8150016.hcp.osadev.cloud": context deadline exceeded
    {
        Op: "Get",
        URL: "https://console-openshift-console.apps.aro.private-kas.0ju5.j8150016.hcp.osadev.cloud",
        Err: <context.deadlineExceededError>{},
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72322110882585688150016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRRevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 1 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: failed to revoke admin credentials for cluster "admin-cred-lifecycle-mjx8hf"
Unexpected error:
    <*fmt.wrapErrors | 0x1ab46da90510>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-mjx8hf" in resourcegroup="admin-credential-lifecycle-test-k6wptwmmb252" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-mjx8hf in resource group admin-credential-lifecycle-test-k6wptwmmb252, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-mjx8hf\" in resourcegroup=\"admin-credential-lifecycle-test-k6wptwmmb252\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-mjx8hf in resource group admin-credential-lifecycle-test-k6wptwmmb252, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1ab46cac9c40>{
                s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-mjx8hf in resource group admin-credential-lifecycle-test-k6wptwmmb252",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:222]: failed to revoke admin credentials for cluster "admin-cred-lifecycle-mjx8hf"
Unexpected error:
    <*fmt.wrapErrors | 0x1ab46da90510>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-mjx8hf" in resourcegroup="admin-credential-lifecycle-test-k6wptwmmb252" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-mjx8hf in resource group admin-credential-lifecycle-test-k6wptwmmb252, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-mjx8hf\" in resourcegroup=\"admin-credential-lifecycle-test-k6wptwmmb252\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-mjx8hf in resource group admin-credential-lifecycle-test-k6wptwmmb252, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1ab46cac9c40>{
                s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-mjx8hf in resource group admin-credential-lifecycle-test-k6wptwmmb252",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72502111378042730319872
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bb...
full failure pattern:
VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff)
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x165441e2cf48>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]
    {
        errs: [
            <*fmt.wrapError | 0x165441dab3a0>{
                msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]",
                err: <*fmt.wrapError | 0x165441dab1c0>{
                    msg: "connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]",
                    err: <*errors.errorString | 0x1654427290b0>{
                        s: "test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]",
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:408]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0x165441e2cf48>: 
    VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]
    {
        errs: [
            <*fmt.wrapError | 0x165441dab3a0>{
                msg: "VerifyCiliumConnectivityChecks failed: connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]",
                err: <*fmt.wrapError | 0x165441dab1c0>{
                    msg: "connectivity check failed, not all pods in cilium-connectivity-check-bhfzdb namespace are ready: test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]",
                    err: <*errors.errorString | 0x1654427290b0>{
                        s: "test pods not yet ready: [pod-to-a-5b5d56bbc9-gjjkg (container pod-to-a-container waiting: CrashLoopBackOff) pod-to-a-allowed-cnp-ddf755dfc-vqspz (container pod-to-a-allowed-cnp-container waiting: CrashLoopBackOff) pod-to-b-multi-node-headless-8656ffdc6-6hrzt (container pod-to-b-multi-node-headless-container waiting: CrashLoopBackOff)]",
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75312111218975433035776
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyCloudNetworkConfigEgressIP(<ip>) failed: EgressIP "e2e-cloud-network-config-test" was never assigned to a node: timed out waiting for EgressIP "e2e-cloud-network-config-test...
full failure pattern:
VerifyCloudNetworkConfigEgressIP(<ip>) failed: EgressIP "e2e-cloud-network-config-test" was never assigned to a node: timed out waiting for EgressIP "e2e-cloud-network-config-test" to be assigned to a node: EgressIP "e2e-cloud-network-config-test" is not yet
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_operators.go:163]: failed to verify cluster operators on cluster "cluster-operators-qwqhtt"
Unexpected error:
    <*errors.joinError | 0x275bb568108>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP "e2e-cloud-network-config-test" was never assigned to a node: timed out waiting for EgressIP "e2e-cloud-network-config-test" to be assigned to a node: EgressIP "e2e-cloud-network-config-test" is not yet assigned to a node
    {
        errs: [
            <*fmt.wrapError | 0x275bb4fe480>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                err: <*fmt.wrapError | 0x275baa78f20>{
                    msg: "EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                    err: <*fmt.wrapError | 0x275baa78d80>{
                        msg: "timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                        err: <*errors.errorString | 0x275bb7584c0>{
                            s: "EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_operators.go:163]: failed to verify cluster operators on cluster "cluster-operators-qwqhtt"
Unexpected error:
    <*errors.joinError | 0x275bb568108>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP "e2e-cloud-network-config-test" was never assigned to a node: timed out waiting for EgressIP "e2e-cloud-network-config-test" to be assigned to a node: EgressIP "e2e-cloud-network-config-test" is not yet assigned to a node
    {
        errs: [
            <*fmt.wrapError | 0x275bb4fe480>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                err: <*fmt.wrapError | 0x275baa78f20>{
                    msg: "EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                    err: <*fmt.wrapError | 0x275baa78d80>{
                        msg: "timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                        err: <*errors.errorString | 0x275bb7584c0>{
                            s: "EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelVerify cluster operators should provision an HCP cluster and verify the cluster operators1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74632109977127079120896
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout
full failure pattern:
VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xb019e0de2e8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xb019c86fe00>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c86fda0>{
                    msg: "failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df811d0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xb019c4d6ff0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019bfab1e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019bfab0e0>{
                    msg: "failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df140c0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xb019c5e2a50>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019c4e0980>{
                msg: "VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c4e0960>{
                    msg: "failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019e2caff0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default",
                        Err: <*net.OpError | 0xb019bfc47d0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xb019e0de2e8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout
    {
        errs: [
            <*fmt.wrapError | 0xb019c86fe00>{
                msg: "VerifyBasicAccess failed: failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c86fda0>{
                    msg: "failed to list services: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df811d0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/api/v1/namespaces/default/services",
                        Err: <*net.OpError | 0xb019c4d6ff0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019bfab1e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019bfab0e0>{
                    msg: "failed to list all APIServices: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019df140c0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/apiregistration.k8s.io/v1/apiservices",
                        Err: <*net.OpError | 0xb019c5e2a50>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xb019c4e0980>{
                msg: "VerifyIngressControllerScope(Internal) failed: failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                err: <*fmt.wrapError | 0xb019c4e0960>{
                    msg: "failed to get default IngressController: Get \"https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default\": dial tcp: lookup api.private-ingress.9krg.j0754432.hcp.osadev.cloud on 172.30.0.10:53: read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                    err: <*url.Error | 0xb019e2caff0>{
                        Op: "Get",
                        URL: "https://api.private-ingress.9krg.j0754432.hcp.osadev.cloud:443/apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default",
                        Err: <*net.OpError | 0xb019bfc47d0>{
                            Op: "dial",
                            Net: "tcp",
                            Source: nil,
                            Addr: nil,
                            Err: <*net.DNSError | 0xb019c4d6eb0>{
                                UnwrapErr: nil,
                                Err: "read udp 172.24.18.106:35378->172.30.0.10:53: i/o timeout",
                                Name: "api.private-ingress.9krg.j0754432.hcp.osadev.cloud",
                                Server: "172.30.0.10:53",
                                IsTimeout: true,
                                IsTemporary: true,
                                IsNotFound: false,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71622109675831700754432
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1f63e25da900>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-15T00:18:45.8286809Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f", ActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", SubActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityType:"GW.Http.CallContext", ParentActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityStack:"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1f63e28703a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1f63e2d64800>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-15T00:18:45.8286809Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f",
                        ActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        SubActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1f63e25da900>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-15T00:18:45.8286809Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f", ActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", SubActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityType:"GW.Http.CallContext", ParentActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityStack:"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1f63e28703a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1f63e2d64800>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-15T00:18:45.8286809Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f",
                        ActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        SubActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74512110505977575903232
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j2678400-mgmt-1/ocm-arohcpci01-<id>: expected
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jv8jpqmq9gpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x239a48fa84c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jv8jpqmq9gpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j2678400-mgmt-1/ocm-arohcpci01-2tft4ek4kogcffngbs1nats12glqcn7n-x6a3m7l5u9v6a9d: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jv8jpqmq9gpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j2678400-mgmt-1/ocm-arohcpci01-2tft4ek4kogcffngbs1nats12glqcn7n-x6a3m7l5u9v6a9d: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
        err: <*errors.errorString | 0x239a4a108640>{
            s: "dev/westus3/ci01-j2678400-mgmt-1/ocm-arohcpci01-2tft4ek4kogcffngbs1nats12glqcn7n-x6a3m7l5u9v6a9d: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jv8jpqmq9gpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x239a48fa84c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jv8jpqmq9gpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j2678400-mgmt-1/ocm-arohcpci01-2tft4ek4kogcffngbs1nats12glqcn7n-x6a3m7l5u9v6a9d: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jv8jpqmq9gpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j2678400-mgmt-1/ocm-arohcpci01-2tft4ek4kogcffngbs1nats12glqcn7n-x6a3m7l5u9v6a9d: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
        err: <*errors.errorString | 0x239a4a108640>{
            s: "dev/westus3/ci01-j2678400-mgmt-1/ocm-arohcpci01-2tft4ek4kogcffngbs1nats12glqcn7n-x6a3m7l5u9v6a9d: expected all current routers >= desired replicas > 0; desired=3 current=0 rows=1",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74892111067237912678400
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2620bace6480>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x2620bace6460>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2620bace6480>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x2620bace6460>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75252110338041116626944
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jn4vtkkhs7mr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3527505f6640>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jn4vtkkhs7mr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jn4vtkkhs7mr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x3527505f6620>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jn4vtkkhs7mr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3527505f6640>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jn4vtkkhs7mr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jn4vtkkhs7mr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x3527505f6620>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74972110558398402531328
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x21de3cbea400>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:29.6322749Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736", ActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", SubActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityStack:"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x21de3ca9e560>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x21de3e2c5500>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:29.6322749Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736",
                        ActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        SubActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x21de3cbea400>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:29.6322749Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736", ActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", SubActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityStack:"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x21de3ca9e560>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x21de3e2c5500>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:29.6322749Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736",
                        ActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        SubActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75222110302904370860032
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j6657920-mgmt-2/ocm-arohcpci01-<id>: expected
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4hk6sg2whd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x21a95a4b0000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4hk6sg2whd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j6657920-mgmt-2/ocm-arohcpci01-2tfp6vu1t0esj9s2f87m52kogvfeb12i-d8h3a4b2v3x4x1h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4hk6sg2whd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j6657920-mgmt-2/ocm-arohcpci01-2tfp6vu1t0esj9s2f87m52kogvfeb12i-d8h3a4b2v3x4x1h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x21a95b98e500>{
            s: "dev/centralus/ci01-j6657920-mgmt-2/ocm-arohcpci01-2tfp6vu1t0esj9s2f87m52kogvfeb12i-d8h3a4b2v3x4x1h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4hk6sg2whd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x21a95a4b0000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4hk6sg2whd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j6657920-mgmt-2/ocm-arohcpci01-2tfp6vu1t0esj9s2f87m52kogvfeb12i-d8h3a4b2v3x4x1h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4hk6sg2whd2t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: dev/centralus/ci01-j6657920-mgmt-2/ocm-arohcpci01-2tfp6vu1t0esj9s2f87m52kogvfeb12i-d8h3a4b2v3x4x1h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x21a95b98e500>{
            s: "dev/centralus/ci01-j6657920-mgmt-2/ocm-arohcpci01-2tfp6vu1t0esj9s2f87m52kogvfeb12i-d8h3a4b2v3x4x1h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75662110999442776657920
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xee59296c020>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0xee59296c000>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xee59296c020>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0xee59296c000>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75232110300965679665152
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9678720-mgmt-1/ocm-arohcpci01-<id>: expected
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lc82txzvgr45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x11ea08e84580>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lc82txzvgr45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9678720-mgmt-1/ocm-arohcpci01-2tfouap5pj715l32pn7smi3ihfg5rglm-y5e3f0u2q1z1x6w: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lc82txzvgr45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9678720-mgmt-1/ocm-arohcpci01-2tfouap5pj715l32pn7smi3ihfg5rglm-y5e3f0u2q1z1x6w: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        err: <*errors.errorString | 0x11ea09d18c20>{
            s: "dev/centralus/ci01-j9678720-mgmt-1/ocm-arohcpci01-2tfouap5pj715l32pn7smi3ihfg5rglm-y5e3f0u2q1z1x6w: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lc82txzvgr45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x11ea08e84580>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lc82txzvgr45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9678720-mgmt-1/ocm-arohcpci01-2tfouap5pj715l32pn7smi3ihfg5rglm-y5e3f0u2q1z1x6w: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lc82txzvgr45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/centralus/ci01-j9678720-mgmt-1/ocm-arohcpci01-2tfouap5pj715l32pn7smi3ihfg5rglm-y5e3f0u2q1z1x6w: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        err: <*errors.errorString | 0x11ea09d18c20>{
            s: "dev/centralus/ci01-j9678720-mgmt-1/ocm-arohcpci01-2tfouap5pj715l32pn7smi3ihfg5rglm-y5e3f0u2q1z1x6w: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74402110993516539678720
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1000320-mgmt-1/ocm-arohcpci01-<id>
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-cdzxpnm9wgtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3222464a4800>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-cdzxpnm9wgtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1000320-mgmt-1/ocm-arohcpci01-2tg0gb0ik8kqufv9aj7d0iha95qjnf3j-e7g2r4n0l4l5r1h router=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000: expected matching summary after test start and within 8m; summaryUID=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000 pass=2b7ade69-c160-46d5-8524-479bdeb90f91 at=2026-10-16T17:09:43Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-cdzxpnm9wgtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1000320-mgmt-1/ocm-arohcpci01-2tg0gb0ik8kqufv9aj7d0iha95qjnf3j-e7g2r4n0l4l5r1h router=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000: expected matching summary after test start and within 8m; summaryUID=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000 pass=2b7ade69-c160-46d5-8524-479bdeb90f91 at=2026-10-16T17:09:43Z",
        err: <*errors.errorString | 0x322245a3b8b0>{
            s: "dev/westus3/ci01-j1000320-mgmt-1/ocm-arohcpci01-2tg0gb0ik8kqufv9aj7d0iha95qjnf3j-e7g2r4n0l4l5r1h router=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000: expected matching summary after test start and within 8m; summaryUID=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000 pass=2b7ade69-c160-46d5-8524-479bdeb90f91 at=2026-10-16T17:09:43Z",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-cdzxpnm9wgtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3222464a4800>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-cdzxpnm9wgtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1000320-mgmt-1/ocm-arohcpci01-2tg0gb0ik8kqufv9aj7d0iha95qjnf3j-e7g2r4n0l4l5r1h router=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000: expected matching summary after test start and within 8m; summaryUID=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000 pass=2b7ade69-c160-46d5-8524-479bdeb90f91 at=2026-10-16T17:09:43Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-cdzxpnm9wgtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1000320-mgmt-1/ocm-arohcpci01-2tg0gb0ik8kqufv9aj7d0iha95qjnf3j-e7g2r4n0l4l5r1h router=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000: expected matching summary after test start and within 8m; summaryUID=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000 pass=2b7ade69-c160-46d5-8524-479bdeb90f91 at=2026-10-16T17:09:43Z",
        err: <*errors.errorString | 0x322245a3b8b0>{
            s: "dev/westus3/ci01-j1000320-mgmt-1/ocm-arohcpci01-2tg0gb0ik8kqufv9aj7d0iha95qjnf3j-e7g2r4n0l4l5r1h router=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000: expected matching summary after test start and within 8m; summaryUID=0fa463c2-2001-466d-92cc-dfa36a320846 node=aks-userswft3-10989426-vmss000000 pass=2b7ade69-c160-46d5-8524-479bdeb90f91 at=2026-10-16T17:09:43Z",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69282111120828451000320
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x298032100120>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x298032100100>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x298032100120>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x298032100100>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74992110452208637054976
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1dded76126c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:43.5550944Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c", ActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", SubActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityType:"GW.Http.CallContext", ParentActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityStack:"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1dded76126a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1dded776cc00>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:43.5550944Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c",
                        ActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        SubActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1dded76126c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:43.5550944Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c", ActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", SubActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityType:"GW.Http.CallContext", ParentActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityStack:"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1dded76126a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1dded776cc00>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:43.5550944Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c",
                        ActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        SubActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73832110303886584582144
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x128374ba040>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x128374ba020>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x128374ba040>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x128374ba020>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74982110305441572458496
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j8670976-mgmt-2/ocm-arohcpci01-<id>: expected all
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w6xbc82hxr2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x280af577a000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w6xbc82hxr2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j8670976-mgmt-2/ocm-arohcpci01-2tf1898kdmei2jv6trajehsfpj8o4bk2-f7r5q8n1l4e4d9o: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w6xbc82hxr2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j8670976-mgmt-2/ocm-arohcpci01-2tf1898kdmei2jv6trajehsfpj8o4bk2-f7r5q8n1l4e4d9o: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        err: <*errors.errorString | 0x280af63caa80>{
            s: "dev/centralus/ci01-j8670976-mgmt-2/ocm-arohcpci01-2tf1898kdmei2jv6trajehsfpj8o4bk2-f7r5q8n1l4e4d9o: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w6xbc82hxr2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x280af577a000>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w6xbc82hxr2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j8670976-mgmt-2/ocm-arohcpci01-2tf1898kdmei2jv6trajehsfpj8o4bk2-f7r5q8n1l4e4d9o: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w6xbc82hxr2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/centralus/ci01-j8670976-mgmt-2/ocm-arohcpci01-2tf1898kdmei2jv6trajehsfpj8o4bk2-f7r5q8n1l4e4d9o: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        err: <*errors.errorString | 0x280af63caa80>{
            s: "dev/centralus/ci01-j8670976-mgmt-2/ocm-arohcpci01-2tf1898kdmei2jv6trajehsfpj8o4bk2-f7r5q8n1l4e4d9o: expected all current routers >= desired replicas > 0; desired=0 current=0 rows=1",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67462110588894188670976
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x32bc856472e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x32bc85647260>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x32bc856472e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x32bc85647260>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67462110452257878183936
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRafter 4 attempts: 1 releases found in <url> which is not enough for the requested 1 offset
full failure pattern:
after 4 attempts: 1 releases found in <url> which is not enough for the requested 1 offset
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:134]: failed to resolve configured channel candidate-5.1
Unexpected error:
    <*fmt.wrapError | 0xda3675c58e0>: 
    after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset
    {
        msg: "after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
        err: <*errors.errorString | 0xda366e3dab0>{
            s: "1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:134]: failed to resolve configured channel candidate-5.1
Unexpected error:
    <*fmt.wrapError | 0xda3675c58e0>: 
    after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset
    {
        msg: "after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
        err: <*errors.errorString | 0xda366e3dab0>{
            s: "1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772110068844331536384
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] PrometheusUptime fired
full failure pattern:
alert [svc] PrometheusUptime fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-15T11:00:07Z
  Ended:       2026-10-15T11:20:06Z
  Severity:    Sev2
  Labels:      alertname="PrometheusUptime", cluster="ci01-j0312192-mgmt-2", component="monitoring-infrastructure", job="prometheus/prometheus", namespace="prometheus", region="centralus", severity="critical"
  Description: Fewer than 95% of present Prometheus self-up samples were healthy over the past 24 hours.
This may indicate that the Prometheus server is down, experiencing network issues, or stuck in a crash loop.
Missing samples are covered separately by PrometheusUptimeSampleCount.
Please check the status of the Prometheus pods, service endpoints, and network connectivity.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert PrometheusUptime does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74672110665048610312192
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] userJourneyClusterDeletionErrors1h5m fired
full failure pattern:
alert [svc] userJourneyClusterDeletionErrors1h5m fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-16T13:12:08Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterDeletionErrors1h5m", cluster="ci01-j9203200-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9203200/providers/microsoft.monitor/accounts/services-j9203200", region="westus3", severity="3", short_window="5m", slo="cluster-deletion-errors"
  Description: More than 72% of completed cluster delete operations on ci01-j9203200-svc failed or were canceled over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert userJourneyClusterDeletionErrors1h5m does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69822111041799249203200
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] userJourneyClusterDeletionErrors6h30m fired
full failure pattern:
alert [svc] userJourneyClusterDeletionErrors6h30m fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-16T12:43:13Z
  Ended:       2026-10-16T13:05:11Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterDeletionErrors6h30m", cluster="ci01-j9203200-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9203200/providers/microsoft.monitor/accounts/services-j9203200", region="westus3", severity="3", short_window="30m", slo="cluster-deletion-errors"
  Description: More than 30% of completed cluster delete operations on ci01-j9203200-svc failed or were canceled over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert userJourneyClusterDeletionErrors6h30m does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69822111041799249203200
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRclusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no regis...
full failure pattern:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04="
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x181b25f73380>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04="
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 \"3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04=\"",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x181b25f73380>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04="
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 \"3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04=\"",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75002110389003348873216
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRerror running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...
full failure pattern:
error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j2981888-mgmt-1","clusterName":"ci01-j298188
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
time=2026-10-14T19:33:50.044Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-14T20:03:49.640Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-14T19:33:50.055321564Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2981888-mgmt-1\",\"clusterName\":\"ci01-j2981888-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-14T19:33:52.298968498Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2981888-mgmt-1\",\"clusterName\":\"ci01-j2981888-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-14T19:33:52.677544841Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2981888-mgmt-1\",\"clusterName\":\"ci01-j2981888-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69462110441834562981888
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRerror running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...
full failure pattern:
error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j8714496-mgmt-1","clusterName":"ci01-j871449
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.20%1 of 491 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0none
Full failure examples (1)
time=2026-10-16T14:04:27.353Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-16T14:34:26.879Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-16T14:04:27.363830736Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8714496-mgmt-1\",\"clusterName\":\"ci01-j8714496-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-16T14:04:29.161200885Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8714496-mgmt-1\",\"clusterName\":\"ci01-j8714496-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-16T14:04:29.579255531Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8714496-mgmt-1\",\"clusterName\":\"ci01-j8714496-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72322111087082628714496