CIHealth
<
Single Day: Oct 18▾
Last 7 Days Weekly: Oct 12 - Oct 18 Sprint: Oct 12 - Oct 25 Single Day: Oct 18
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 5 · Failures matched: 7 · Total runs: 11 · Runs affected: 7

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key "globalMSIId" not found
full failure pattern:
failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key "globalMSIId" not found
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision218.18%2 of 11 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 4 · Oct 17: 0 · Oct 18: 2none
Full failure examples (2)
time=2026-10-18T15:38:05.575Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-18T15:38:06.914Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found"
time=2026-10-18T12:10:56.819Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-18T12:10:57.618Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #75802111837676591124480
PR #75802111776757118930944
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
alert218.18%2 of 11 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsOct 12: 5 · Oct 13: 6 · Oct 14: 4 · Oct 15: 10 · Oct 16: 9 · Oct 17: 1 · Oct 18: 2none
Full failure examples (2)
alert fired 38 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-tmw4b", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-tmw4b has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-4fdv6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-4fdv6 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-5vc4f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-5vc4f has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-dfx48", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-dfx48 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-tmkfh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-tmkfh has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-ldq7g", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-ldq7g has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-4prhm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-4prhm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-mtvd8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-mtvd8 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-w9lfx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-w9lfx has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-m6kzd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-m6kzd has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-gf4mg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-gf4mg has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-jtg79", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-jtg79 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-gbc9d", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-gbc9d has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-vnwdn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-vnwdn has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-cvdcq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-cvdcq has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-jp4zq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-jp4zq has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-lcvwv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-lcvwv has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-nllhh", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-nllhh has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-x8lvp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-x8lvp has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-2cp2r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-2cp2r has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-bnghx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-bnghx has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-spjrm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-spjrm has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-7gwqf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-7gwqf has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-wzhd4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-wzhd4 has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-z67pp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-z67pp has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-18T22:30:02Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-8rjbk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-8rjbk has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-89fp5", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-89fp5 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-dfhsg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-dfhsg has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-ndv7w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-ndv7w has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-x65xf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-x65xf has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-9nhcz", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-9nhcz has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-57rg5", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-57rg5 has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-vqfkk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-vqfkk has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-mvbzv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-mvbzv has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-xqzzm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-xqzzm has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-pzz7r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-pzz7r has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-4ht2v", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-4ht2v has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-18T22:31:04Z
  Ended:       2026-10-18T22:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-m2n57", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-m2n57 has been in a non-ready state for longer than 5 minutes.
alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-18T02:22:20Z
  Ended:       2026-10-18T02:28:23Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5039232-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgtj1aja7eu3kumpui0v9v0hrn0ctrn-n2t2k5k1d2q7b4m", pod="openshift-apiserver-74747bbdf5-l6sx2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2tgtj1aja7eu3kumpui0v9v0hrn0ctrn-n2t2k5k1d2q7b4m/openshift-apiserver-74747bbdf5-l6sx2 has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73872111927716537176064
PR #73822111625789635039232
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j1046272-mgmt-1/ocm-arohcpci01-<id>
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e19.09%1 of 11 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-v9j4v95cl9wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xadaa6040100>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-v9j4v95cl9wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j1046272-mgmt-1/ocm-arohcpci01-2th7q5a54cq5jldof0f3l7iofpg7u8al-h1p2a3c0s8o4u6j: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-v9j4v95cl9wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j1046272-mgmt-1/ocm-arohcpci01-2th7q5a54cq5jldof0f3l7iofpg7u8al-h1p2a3c0s8o4u6j: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0xadaa51438f0>{
            s: "dev/canadacentral/ci01-j1046272-mgmt-1/ocm-arohcpci01-2th7q5a54cq5jldof0f3l7iofpg7u8al-h1p2a3c0s8o4u6j: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-v9j4v95cl9wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xadaa6040100>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-v9j4v95cl9wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j1046272-mgmt-1/ocm-arohcpci01-2th7q5a54cq5jldof0f3l7iofpg7u8al-h1p2a3c0s8o4u6j: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-v9j4v95cl9wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/canadacentral/ci01-j1046272-mgmt-1/ocm-arohcpci01-2th7q5a54cq5jldof0f3l7iofpg7u8al-h1p2a3c0s8o4u6j: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0xadaa51438f0>{
            s: "dev/canadacentral/ci01-j1046272-mgmt-1/ocm-arohcpci01-2th7q5a54cq5jldof0f3l7iofpg7u8al-h1p2a3c0s8o4u6j: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75912111797770821046272
failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded
full failure pattern:
failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
provision19.09%1 of 11 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 1none
Full failure examples (1)
time=2026-10-18T21:53:08.675Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-18T21:53:10.010Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3670528-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-18T21:53:10.010Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3670528-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-18T21:53:10.816Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3670528-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-18T21:53:10.816Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j3670528-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-18T21:53:10.816Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j3670528-svc
time=2026-10-18T22:08:13.794Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75802111930721403670528
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDB
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
provision19.09%1 of 11 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 1none
Full failure examples (1)
time=2026-10-18T22:01:09.062Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-18T22:01:10.583Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-18T22:01:11.299Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d2f2367984b48d226a7d218c1b893f5e8345defcea4da99b6318b3f53d90ed49 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6019200%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd2f2367984b48d226a7d218c1b893f5e8345defcea4da99b6318b3f53d90ed49
time=2026-10-18T22:08:43.317Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6019200/providers/Microsoft.Resources/deployments/d2f2367984b48d226a7d218c1b893f5e8345defcea4da99b6318b3f53d90ed49/operationStatuses/08584092460146839094\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6019200/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6019200/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j6019200\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n                \\\"message\\\": \\\"Database account creation failed. Operation Id: 29cb1d84-68fc-41bc-85ce-f7a058c770ca, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: 510ce633-a87c-4347-9741-f3200e8ace7f, Request URI: /apps/afdfb817-7910-4985-b158-590770c1c1bf/services/f1fdfe14-2eef-4dca-b710-c2c6e8884c3f/partitions/918f52c2-0d35-4150-826f-7edd51cc07ef/replicas/134367860023629100s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-10-18T22:07:00.4462358Z, RequestEndTime: 2026-10-18T22:07:05.4461411Z,  Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-18T22:06:12.0209020Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.273,\\\\\\\"memory\\\\\\\":20915668.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0851,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-18T22:06:22.0359077Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.546,\\\\\\\"memory\\\\\\\":20914336.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0463,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-18T22:06:32.0513471Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.449,\\\\\\\"memory\\\\\\\":20922484.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0842,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-18T22:06:42.0523437Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.762,\\\\\\\"memory\\\\\\\":20935348.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.083,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-18T22:06:52.0699478Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.390,\\\\\\\"memory\\\\\\\":20926436.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0518,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-10-18T22:07:02.0711735Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.332,\\\\\\\"memory\\\\\\\":20925148.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0848,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":32}]}\\\\r\\\\nRequestStart: 2026-10-18T22:07:00.4462358Z; ResponseTime: 2026-10-18T22:07:05.4461411Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-centralus1-be424.documents.azure.com:11907/apps/afdfb817-7910-4985-b158-590770c1c1bf/services/f1fdfe14-2eef-4dca-b710-c2c6e8884c3f/partitions/918f52c2-0d35-4150-826f-7edd51cc07ef/replicas/134367860023629100s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 4996.861, ActivityId: 510ce633-a87c-4347-9741-f3200e8ace7f, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11907 | status: Connected | lkt: 10/18/2026 10:07:05 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-18T22:07:00.4462358Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0024},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-18T22:07:00.4462382Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0078},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-18T22:07:00.4462460Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0881},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-18T22:07:00.4463341Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 4997.4559},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-18T22:07:05.4437900Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0577},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-10-18T22:07:05.4438477Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-10-18T22:04:46.4616163Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-10-18T22:04:46.4616163Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-10-18T22:04:52.4460367Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":867,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75922111935962996019200