<
Single Day: Oct 11▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 20m 27s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head cb92dba · merge cb92dba
westus3 alert, e2e failed ★#7439 (merged) 2
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
23:29:37 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1b569c9d54e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlcxj9z6srvl" to finish creating: GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04",
      "name": "5a974196-cc3e-4c19-83bb-20557d537e04",
      "status": "Failed",
      "startTime": "2026-10-12T00:41:24.479383887Z",
      "endTime": "2026-10-12T01:10:29.269134441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
            correlation ID: 2bffc39b-cd83-4954-a0a9-c7daa45e4979
            client request ID: 8e77b332-cff0-4849-90bf-1e7ad363de44
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1b569c9d54e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlcxj9z6srvl" to finish creating: GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04",
      "name": "5a974196-cc3e-4c19-83bb-20557d537e04",
      "status": "Failed",
      "startTime": "2026-10-12T00:41:24.479383887Z",
      "endTime": "2026-10-12T01:10:29.269134441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
            correlation ID: 2bffc39b-cd83-4954-a0a9-c7daa45e4979
            client request ID: 8e77b332-cff0-4849-90bf-1e7ad363de44
occurred
alert (1)
alert [hcp] KubePodNotReady fired
23:29:37 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 17 time(s) (14 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-lhgsl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-lhgsl has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-tt7dl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-tt7dl has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:03:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i", pod="ignition-server-proxy-7f44d8d46-qdz9k", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i/ignition-server-proxy-7f44d8d46-qdz9k has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-vp2rh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-vp2rh has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="kube-controller-manager-6fbbbcbff9-8w5vs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/kube-controller-manager-6fbbbcbff9-8w5vs has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-dfpqp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-dfpqp has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm44q5ep2osnuu42ndqjo11p7p3m6-q6t1s1d8h2q6s5k", pod="ovnkube-control-plane-c5d7b9d44-sm4sw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm44q5ep2osnuu42ndqjo11p7p3m6-q6t1s1d8h2q6s5k/ovnkube-control-plane-c5d7b9d44-sm4sw has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-5jm6z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-5jm6z has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-hrjrj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-hrjrj has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:03:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i", pod="ignition-server-proxy-7f44d8d46-8qzfv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i/ignition-server-proxy-7f44d8d46-8qzfv has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-hp7l6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-hp7l6 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-jzrt5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-jzrt5 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-lvr9j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-lvr9j has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm3l4c9fvor23h7j8oq4g14n444ug-ingress-cert-le", pod="kube-controller-manager-bf5f57fb8-bdk2x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm3l4c9fvor23h7j8oq4g14n444ug-ingress-cert-le/kube-controller-manager-bf5f57fb8-bdk2x has been in a non-ready state for longer than 5 minutes.
2h 22m 22s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c154328 · merge c154328
centralus alert, e2e failed #7424 (merged) 2
e2e (1)
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
23:05:50 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zhszcgnzg7zb/cluster-candidate-4-22-gwkf7b should provision
Unexpected error:
    <*fmt.wrapError | 0x17457dc5e0e0>: 
    failed to create HCP cluster cluster-candidate-4-22-gwkf7b: failed waiting for cluster="cluster-candidate-4-22-gwkf7b" in resourcegroup="rg-candidate-4-22-zhszcgnzg7zb" to finish creating: GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
            correlation ID: e870b595-733c-4e50-b159-c06646eb16ba
            client request ID: 7c347d72-8099-4796-893d-59762507c230
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zhszcgnzg7zb/cluster-candidate-4-22-gwkf7b should provision
Unexpected error:
    <*fmt.wrapError | 0x17457dc5e0e0>: 
    failed to create HCP cluster cluster-candidate-4-22-gwkf7b: failed waiting for cluster="cluster-candidate-4-22-gwkf7b" in resourcegroup="rg-candidate-4-22-zhszcgnzg7zb" to finish creating: GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
            correlation ID: e870b595-733c-4e50-b159-c06646eb16ba
            client request ID: 7c347d72-8099-4796-893d-59762507c230
occurred
alert (1)
alert [hcp] KubePodNotReady fired
23:05:50 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T00:36:32Z
  Ended:       2026-10-12T00:41:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8959360-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctccghfgvuis1e4ubo7a7ap2uim0kd-w0e2c0f4m6k0z7f", pod="cluster-network-operator-67f4c84888-hzql4", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tctccghfgvuis1e4ubo7a7ap2uim0kd-w0e2c0f4m6k0z7f/cluster-network-operator-67f4c84888-hzql4 has been in a non-ready state for longer than 5 minutes.
2h 19m 31s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head eebaf98 · merge bf3311f
westus3 alert failed #7162 (merged) 1
alert (1)
alert [svc] LeaderElectionLeaseStale fired
21:12:37 UTC · failed at alert · test [aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-11T23:00:46Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j3354496-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.245:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3354496/providers/microsoft.monitor/accounts/services-j3354496", namespace="klusterlet-2tcrhmibf7k7vglj9iommmu4f595r7e7", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jhnzl", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tcrhmibf7k7vglj9iommmu4f595r7e7 on cluster ci01-j3354496-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
30m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5e5ec5c · merge 5e5ec5c
n/a unknown failed #7458 (merged) 1 non-artifact-backed failure (no junit artifacts)
2h 44m 1s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0c8ab0b · merge 0c8ab0b
westus3 alert, e2e failed #7457 (merged) 2
e2e (1)
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
19:48:48 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.006s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23a94d2a72e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.006s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23a94d2a72e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
alert (1)
alert [hcp] KubePodNotReady fired
19:48:48 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T21:23:26Z
  Ended:       2026-10-11T21:29:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6435200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcqfah109q9u6g2h4huthue530shv73-p6r1u7s0v8h7i2h", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcqfah109q9u6g2h4huthue530shv73-p6r1u7s0v8h7i2h/etcd-0 has been in a non-ready state for longer than 5 minutes.
2h 20m 38s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0c8ab0b · merge 0c8ab0b
centralus alert failed #7457 (merged) 1
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
17:11:29 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T18:33:37Z
  Ended:       2026-10-11T18:42:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8107776-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 23m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a7588ae · merge d13cfae
centralus e2e failed ⚠#6746 (merged) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-<id> to exist; provider Microsoft.RedHatOpenShift
15:50:27 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x2952d6db0d20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-ksddfkp5xbq5" to finish creating: GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "name": "6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "status": "Failed",
      "startTime": "2026-10-11T16:48:04.203625548Z",
      "endTime": "2026-10-11T17:17:06.363062447Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkw9f-sxlv4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkw9f-sxlv4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
            correlation ID: 9211dc6b-1604-468e-b90e-8381473929df
            client request ID: 967f10bc-a33e-4e81-877c-ed8ff3004e75
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x2952d6db0d20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-ksddfkp5xbq5" to finish creating: GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "name": "6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "status": "Failed",
      "startTime": "2026-10-11T16:48:04.203625548Z",
      "endTime": "2026-10-11T17:17:06.363062447Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkw9f-sxlv4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkw9f-sxlv4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
            correlation ID: 9211dc6b-1604-468e-b90e-8381473929df
            client request ID: 967f10bc-a33e-4e81-877c-ed8ff3004e75
occurred
2h 34m 34s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0c8ab0b · merge 0c8ab0b
centralus alert failed #7457 (merged) 1
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
14:35:05 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T15:52:40Z
  Ended:       2026-10-11T16:12:42Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1984128-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 9m 24s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0e1c991
centralus e2e failed #7219 (open) 1
e2e (1)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
12:00:15 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xe1c86824ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-cnc5f6qgdgz8" to finish creating: GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932",
      "name": "58fb2269-9049-4fd2-9274-ac2190a19932",
      "status": "Failed",
      "startTime": "2026-10-11T12:57:29.816928878Z",
      "endTime": "2026-10-11T13:26:38.515811706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xr7ss-282dw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xr7ss-282dw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
            correlation ID: b9b2478c-6a44-4ebb-ade0-4de59279ccb4
            client request ID: 4eee6950-aeb4-430b-b956-66f798ff6c28
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xe1c86824ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-cnc5f6qgdgz8" to finish creating: GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932",
      "name": "58fb2269-9049-4fd2-9274-ac2190a19932",
      "status": "Failed",
      "startTime": "2026-10-11T12:57:29.816928878Z",
      "endTime": "2026-10-11T13:26:38.515811706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xr7ss-282dw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xr7ss-282dw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
            correlation ID: b9b2478c-6a44-4ebb-ade0-4de59279ccb4
            client request ID: 4eee6950-aeb4-430b-b956-66f798ff6c28
2h 9m 21s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 839b3d3
westus3 alert failed #6452 (open) 1
alert (1)
alert [hcp] KubePodNotReady fired
04:32:49 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 119 time(s) (91 unknown, 28 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-s44bv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-s44bv has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-tspd2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-tspd2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-ztqvn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-ztqvn has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-pp7zn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-pp7zn has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-r72lv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-r72lv has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-nqrl7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-nqrl7 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-sx6j8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-sx6j8 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-vlbv7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-vlbv7 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-hjrmd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-hjrmd has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-p4d2c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-p4d2c has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-59nvf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-59nvf has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-p7kmj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-p7kmj has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="control-plane-operator-5d494df9db-kr48l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/control-plane-operator-5d494df9db-kr48l has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-xp74t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-xp74t has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-knbx9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-knbx9 has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-5kklq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-5kklq has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:03:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-vqpjv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-vqpjv has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-764cz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-764cz has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-c2hxl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-c2hxl has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-k7cqj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-k7cqj has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-fkpms", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-fkpms has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-qkvcs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-qkvcs has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-5jxf6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-5jxf6 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-dz5nh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-dz5nh has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-ktmkd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-ktmkd has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-c2lp7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-c2lp7 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-g6bw4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-g6bw4 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-999w8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-999w8 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-qxvsl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-qxvsl has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-dln2j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-dln2j has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-v8flq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-v8flq has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-7r6qc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-7r6qc has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-lj7wd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-lj7wd has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:03:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-vwd26", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-vwd26 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-ph8gv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-ph8gv has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-kh5zr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-kh5zr has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-6r5ln", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-6r5ln has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-xrh2q", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-xrh2q has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-rmgkv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-rmgkv has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-4bd5r", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-4bd5r has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-ztg86", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-ztg86 has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-4twsj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-4twsj has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-bpf5r", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-bpf5r has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-wgc2x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-wgc2x has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-cdkhc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-cdkhc has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-vgll5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-vgll5 has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-ln4zj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-ln4zj has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-7v5qb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-7v5qb has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-4q9wz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-4q9wz has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-rxd9d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-rxd9d has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-6mwn2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-6mwn2 has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-247jg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-247jg has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="control-plane-operator-5d494df9db-nl4zs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/control-plane-operator-5d494df9db-nl4zs has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-fw9c4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-fw9c4 has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-lfq89", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-lfq89 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-v4jn2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-v4jn2 has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-kv5s2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-kv5s2 has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-ddfxc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-ddfxc has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-blff2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-blff2 has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-jffgt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-jffgt has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-mpgpp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-mpgpp has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-wh54t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-wh54t has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-bv7qc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-bv7qc has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-2xgbl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-2xgbl has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-8bvfh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-8bvfh has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-lmvqp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-lmvqp has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-czzdk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-czzdk has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-8lkjj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-8lkjj has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-ww4c7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-ww4c7 has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-qrpjz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-qrpjz has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-568rf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-568rf has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-jt8qn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-jt8qn has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-5tr5d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-5tr5d has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-cpkmb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-cpkmb has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-7ctwg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-7ctwg has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-sb8km", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-sb8km has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-njj8b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-njj8b has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-s57sq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-s57sq has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-svt9x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-svt9x has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:01:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-fngz6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-fngz6 has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-7grwx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-7grwx has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-z56x9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-z56x9 has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-ph8p9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-ph8p9 has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-nplt8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-nplt8 has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-ht56m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-ht56m has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-766ms", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-766ms has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:02:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-bmp96", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-bmp96 has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:03:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a", pod="cluster-api-758886dcdd-tdcb8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdt5p5krgk8c9s2qa52muf0s2leo-o5c6e9o7o9u8u2a/cluster-api-758886dcdd-tdcb8 has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T06:06:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q", pod="ovnkube-control-plane-84746c8bb7-7k2zp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctg51ab7lhn1327cqn1ro610k7o09-g7m4x8h2o0n2u0q/ovnkube-control-plane-84746c8bb7-7k2zp has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:57:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p", pod="network-node-identity-699479db79-hc79n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcctdihs6bopliddcb2r3r2grv2jd6r-f0j1t3p3q3h3e5p/network-node-identity-699479db79-hc79n has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Resolved
  Started:     2026-10-11T05:38:35Z
  Ended:       2026-10-11T05:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9177088-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s", pod="kube-controller-manager-7d88d8ddc6-wdfkq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcct8i6p9rmb6p0qtjmra4rleu9p1p6-y7r8u9b9d9y6s8s/kube-controller-manager-7d88d8ddc6-wdfkq has been in a non-ready state for longer than 5 minutes.