CIHealth
<
Single Day: Oct 11▾
Last 7 Days Weekly: Oct 5 - Oct 11 Sprint: Sep 28 - Oct 11 Single Day: Oct 11
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 8 · Failures matched: 12 · Total runs: 10 · Runs affected: 9

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-<id> to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e110.00%1 of 10 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x2952d6db0d20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-ksddfkp5xbq5" to finish creating: GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "name": "6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "status": "Failed",
      "startTime": "2026-10-11T16:48:04.203625548Z",
      "endTime": "2026-10-11T17:17:06.363062447Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkw9f-sxlv4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkw9f-sxlv4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
            correlation ID: 9211dc6b-1604-468e-b90e-8381473929df
            client request ID: 967f10bc-a33e-4e81-877c-ed8ff3004e75
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x2952d6db0d20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-ksddfkp5xbq5" to finish creating: GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "name": "6fd0636d-925f-45c9-a7fc-b6aa390b633a",
      "status": "Failed",
      "startTime": "2026-10-11T16:48:04.203625548Z",
      "endTime": "2026-10-11T17:17:06.363062447Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-bkw9f-sxlv4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-bkw9f-sxlv4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-ksddfkp5xbq5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-bkw9f-sxlv4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8626176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fd0636d-925f-45c9-a7fc-b6aa390b633a
            correlation ID: 9211dc6b-1604-468e-b90e-8381473929df
            client request ID: 967f10bc-a33e-4e81-877c-ed8ff3004e75
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67462109310669768626176
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days
alert440.00%4 of 10 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 7 daysShow trend detailsOct 5: 3 · Oct 6: 6 · Oct 7: 9 · Oct 8: 3 · Oct 9: 2 · Oct 10: 1 · Oct 11: 4none
Full failure examples (3)
alert fired 17 time(s) (14 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-lhgsl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-lhgsl has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-tt7dl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-tt7dl has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:03:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i", pod="ignition-server-proxy-7f44d8d46-qdz9k", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i/ignition-server-proxy-7f44d8d46-qdz9k has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-vp2rh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-vp2rh has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="kube-controller-manager-6fbbbcbff9-8w5vs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/kube-controller-manager-6fbbbcbff9-8w5vs has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-dfpqp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-dfpqp has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:05:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm44q5ep2osnuu42ndqjo11p7p3m6-q6t1s1d8h2q6s5k", pod="ovnkube-control-plane-c5d7b9d44-sm4sw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm44q5ep2osnuu42ndqjo11p7p3m6-q6t1s1d8h2q6s5k/ovnkube-control-plane-c5d7b9d44-sm4sw has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-5jm6z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-5jm6z has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-hrjrj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-hrjrj has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:03:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i", pod="ignition-server-proxy-7f44d8d46-8qzfv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm576q05rv6mv1tq3kk25is17j9mp-d9q4e1z1g2p8n7i/ignition-server-proxy-7f44d8d46-8qzfv has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-hp7l6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-hp7l6 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:04:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-jzrt5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-jzrt5 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas", pod="ignition-server-proxy-5988489558-lvr9j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm2ma2es3e2pjmur7r5mim45j0fib-private-kas/ignition-server-proxy-5988489558-lvr9j has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-12T00:47:36Z
  Ended:       2026-10-12T01:08:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6174848-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctm3l4c9fvor23h7j8oq4g14n444ug-ingress-cert-le", pod="kube-controller-manager-bf5f57fb8-bdk2x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tctm3l4c9fvor23h7j8oq4g14n444ug-ingress-cert-le/kube-controller-manager-bf5f57fb8-bdk2x has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-12T00:36:32Z
  Ended:       2026-10-12T00:41:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8959360-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tctccghfgvuis1e4ubo7a7ap2uim0kd-w0e2c0f4m6k0z7f", pod="cluster-network-operator-67f4c84888-hzql4", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tctccghfgvuis1e4ubo7a7ap2uim0kd-w0e2c0f4m6k0z7f/cluster-network-operator-67f4c84888-hzql4 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T21:23:26Z
  Ended:       2026-10-11T21:29:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6435200-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tcqfah109q9u6g2h4huthue530shv73-p6r1u7s0v8h7i2h", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tcqfah109q9u6g2h4huthue530shv73-p6r1u7s0v8h7i2h/etcd-0 has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #74392109426221556174848
PR #74242109420238498959360
PR #74572109370651696435200
PR #64522109140137769177088
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
alert220.00%2 of 10 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsOct 5: 26 · Oct 6: 15 · Oct 7: 29 · Oct 8: 19 · Oct 9: 4 · Oct 10: 0 · Oct 11: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T18:33:37Z
  Ended:       2026-10-11T18:42:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8107776-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-11T15:52:40Z
  Ended:       2026-10-11T16:12:42Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1984128-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74572109331063628107776
PR #74572109291724151984128
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e110.00%1 of 10 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsOct 5: 1 · Oct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.006s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23a94d2a72e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.006s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x23a94d2a72e0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74572109370651696435200
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e110.00%1 of 10 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsOct 5: 1 · Oct 6: 1 · Oct 7: 0 · Oct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zhszcgnzg7zb/cluster-candidate-4-22-gwkf7b should provision
Unexpected error:
    <*fmt.wrapError | 0x17457dc5e0e0>: 
    failed to create HCP cluster cluster-candidate-4-22-gwkf7b: failed waiting for cluster="cluster-candidate-4-22-gwkf7b" in resourcegroup="rg-candidate-4-22-zhszcgnzg7zb" to finish creating: GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
            correlation ID: e870b595-733c-4e50-b159-c06646eb16ba
            client request ID: 7c347d72-8099-4796-893d-59762507c230
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-zhszcgnzg7zb/cluster-candidate-4-22-gwkf7b should provision
Unexpected error:
    <*fmt.wrapError | 0x17457dc5e0e0>: 
    failed to create HCP cluster cluster-candidate-4-22-gwkf7b: failed waiting for cluster="cluster-candidate-4-22-gwkf7b" in resourcegroup="rg-candidate-4-22-zhszcgnzg7zb" to finish creating: GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8959360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/48785925-6e9d-4c48-aae6-599ebf8c4425
            correlation ID: e870b595-733c-4e50-b159-c06646eb16ba
            client request ID: 7c347d72-8099-4796-893d-59762507c230
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74242109420238498959360
alert [svc] LeaderElectionLeaseStale fired
full failure pattern:
alert [svc] LeaderElectionLeaseStale fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
alert110.00%1 of 10 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsOct 5: 5 · Oct 6: 6 · Oct 7: 3 · Oct 8: 6 · Oct 9: 2 · Oct 10: 1 · Oct 11: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-11T23:00:46Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j3354496-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.245:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3354496/providers/microsoft.monitor/accounts/services-j3354496", namespace="klusterlet-2tcrhmibf7k7vglj9iommmu4f595r7e7", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jhnzl", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tcrhmibf7k7vglj9iommmu4f595r7e7 on cluster ci01-j3354496-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71622109391747573354496
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 7 day(s) — New failure pattern — no prior history
e2e110.00%1 of 10 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 7 day(s) — New failure pattern — no prior historyShow trend detailsOct 5: 1 · Oct 6: 2 · Oct 7: 1 · Oct 8: 2 · Oct 9: 1 · Oct 10: 1 · Oct 11: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1b569c9d54e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlcxj9z6srvl" to finish creating: GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04",
      "name": "5a974196-cc3e-4c19-83bb-20557d537e04",
      "status": "Failed",
      "startTime": "2026-10-12T00:41:24.479383887Z",
      "endTime": "2026-10-12T01:10:29.269134441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
            correlation ID: 2bffc39b-cd83-4954-a0a9-c7daa45e4979
            client request ID: 8e77b332-cff0-4849-90bf-1e7ad363de44
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1b569c9d54e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mlcxj9z6srvl" to finish creating: GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04",
      "name": "5a974196-cc3e-4c19-83bb-20557d537e04",
      "status": "Failed",
      "startTime": "2026-10-12T00:41:24.479383887Z",
      "endTime": "2026-10-12T01:10:29.269134441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6174848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a974196-cc3e-4c19-83bb-20557d537e04
            correlation ID: 2bffc39b-cd83-4954-a0a9-c7daa45e4979
            client request ID: 8e77b332-cff0-4849-90bf-1e7ad363de44
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74392109426221556174848
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e110.00%1 of 10 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xe1c86824ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-cnc5f6qgdgz8" to finish creating: GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932",
      "name": "58fb2269-9049-4fd2-9274-ac2190a19932",
      "status": "Failed",
      "startTime": "2026-10-11T12:57:29.816928878Z",
      "endTime": "2026-10-11T13:26:38.515811706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xr7ss-282dw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xr7ss-282dw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
            correlation ID: b9b2478c-6a44-4ebb-ade0-4de59279ccb4
            client request ID: 4eee6950-aeb4-430b-b956-66f798ff6c28
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:137]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xe1c86824ac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-cnc5f6qgdgz8" to finish creating: GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932",
      "name": "58fb2269-9049-4fd2-9274-ac2190a19932",
      "status": "Failed",
      "startTime": "2026-10-11T12:57:29.816928878Z",
      "endTime": "2026-10-11T13:26:38.515811706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xr7ss-282dw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xr7ss-282dw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-cnc5f6qgdgz8--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xr7ss-282dw to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5701760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/58fb2269-9049-4fd2-9274-ac2190a19932
            correlation ID: b9b2478c-6a44-4ebb-ade0-4de59279ccb4
            client request ID: 4eee6950-aeb4-430b-b956-66f798ff6c28
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72192109252833235701760